Some benchmark results

Discussion in 'Mac Programming' started by Graham Perks, Jun 21, 2017.

  1. Graham Perks macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #1
    Here are some build times of a proprietary project on a variety of Macs. If you can suggest a easy-to-build open source project that doesn't have pre-reqs I can re-run with that and compare with other people.

    Killed other apps, ensured Time Machine wasn't about to run. Average of runs of
    rm -rf DerivedData &&
    time xcodebuild -workspace SnapKitchen.xcworkspace -scheme SnapKitchen -configuration Debug

    I let things cool to < 50ºC before re-running, which didn't take long.

    MacBook early 2016, 1.2GHz m5 8GB RAM (500GB SSD) Time: 4m:45s
    MBP 15” 2015 2.2GHz i7 16GB: 2m:08s
    iMac 27” 5K late 2015 8GB 3.3GHz i5: 2m:03s

    Of course the 2nd two are much faster, being quad-core machines with fans.

    Using Intel's Power Gadget I was able to watch the CPU boost speed. The MacBook started around 2.4 GHz then after a minute or so reached 95º (ish) and dropped back to around 2.0Ghz. Used around 8W. The MBP mostly ran at 3.2GHz, used 50W. iMac: around 3.6GHz and 42W.

    The iMac feels faster so I was surprised it wasn't actually much faster that the MBP. Perhaps the larger screen wins me over; or perhaps the 8GB is constraining?

    While this is a full project build, which isn't something you run that frequently; it gives a comparison.

    The iMac is wonderful to use. The MacBook is also wonderful, on the lap. The MBP is a little heavy for a lap and small for a desktop. it's probably my least favorite of the three. The newer lighter MBPs would surely rank higher on my highly personal, subjective "like" scale. On a desk the iMac wins (screen size!). On your lap the lightness of the MacBook wins. All 3 are excellent and differences are quibbles :)
     
  2. Senor Cuete macrumors regular

    Joined:
    Nov 9, 2011
    #2
    I'm responsible for a good-sized project that is a Mac application on the App Store. I develop this on a late 2015 iMac with a 3.1 GHz Intel Core i5. The project has a lot of .xibs, panels and illustrations, a help book, etc. If I clean and rebuild the project it takes about six seconds to build. Of course when working on it XCode only compiles and links the edited files. I can't imagine what sort of project you would be working on that would take that long to build.
     
  3. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #3
    I'd love to find out I have a poor compiler flag somewhere that is causing a slowdown. It's just gradually got slower over time though. I have run the option that shows any particularly bad lines that the Swift parser is choking on.

    This is an iOS app with a handful of Cocoapods. Outside the pods, there are 175 Swift files, 105 XIBs, and 30 storyboards.

    After six seconds my project is still working through the Pods (Alamofire, SwiftyJSON, Fabric, Stripe, and a few for ad trackers/analytics.

    A clean & build from within Xcode is much faster (57 seconds on the iMac) - I believe because Xcode's build normally only builds for one architecture.
    --- Post Merged, Jun 21, 2017 ---
    Upshot is, unless you have an unusually large project, you don't need a Pro machine. When getting started a MacBook is a perfectly excellent choice.
     
  4. Krevnik macrumors 68040

    Krevnik

    Joined:
    Sep 8, 2003
    #4
    I can't say too much about what I've picked up working in a shop with much longer build times, but I can say a lot of it depends on where you are spending the time during build. We did realize that linking is pretty expensive, and hurts incremental builds the most.

    Linking is something that demands more of the disk and isn't something a quad core helps you with, and will tend to drag faster machines down towards the speed of the disk. So the more often you invoke the linker, the more that speedy CPU goes to waste.

    Cocoapods in this case is a great example. Each one of those needs to be pushed through a bit of a linking process, and then linked against your main app. Even with static linking, this is a cost. And while linking, that target won't be using a lot of CPU, as it is more I/O bound. This means that having more targets, especially with a linear dependency chain will slow down your builds too, since you will have a situation where Target A is linking, but Target B is using the idle cores much less frequently.
    --- Post Merged, Jun 21, 2017 ---
    I do agree with this. We definitely fit into the unusually large project category. But I can go home and do smaller projects on an iMac and feel like the thing is too fast compared to what I'm used to.
     
  5. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #5
    Thanks Krevnik, some great thoughts there. Have you any pearls of wisdom for reducing/eliminating link time?

    Usually if I touch some source and Cmd-R, the Cocoapods are not rebuilt of course. A good chunk of the edit-test time is actually post-build - attaching to the simulator etc. That seems to be a few seconds of fixed overhead :(
     
  6. Krevnik macrumors 68040

    Krevnik

    Joined:
    Sep 8, 2003
    #6
    There's really only a couple ways to do it:

    1) Reduce the amount of work the linker has to do. Fewer targets help a bit here.
    2) Faster drives. SSDs instead of HDDs, faster PCIe SSDs instead of SATA ones.

    But the truth is that you are working in a project of a couple hundred files. Linking will take time, and will sometimes be the long pole. It depends on your exact codebase, how it is sliced up, etc. Gains are also somewhat small. In your case, I'd expect the best case gain in a clean build would be on the order of about 12 seconds or so. Or about 6 seconds each flavor.

    And in my opinion, a few seconds isn't a big enough deal on clean builds to worry about. Especially if your incremental builds are fast and reliable. I'd love it if my single-flavor incremental builds were as fast and reliable as your clean builds.

    The bigger the project gets, the longer it will take to build. It's hard to get around that particular problem.
     
  7. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #7
    It will depend on the source language.

    I read a brief something a while ago that showed the Swift compiler is vastly slower than the C or Obj-C compilers, when compiling similar amounts of code, with similar levels of complexity. I think it was at least 10x slower, maybe even more. I don't know if I could even find the blog post again, as it was months ago.

    Personally, I would not be at all surprised if an Obj-C app compiled in 6 seconds. Especially on an 8-core machine, or with an SSD, or some such configuration.


    If you're looking for ways to speed things up, have you googled for things like swift compiler speed?

    Or more generally, What Have You Tried?
     
  8. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #8
    Really I was just posting some times as people often ask "Is a MacBook enough?" "Is a faster CPU worth it?" etc. Just putting out some Xcode numbers as Xcode is rarely, if ever, shown in review benchmarks.
     
  9. Senor Cuete macrumors regular

    Joined:
    Nov 9, 2011
    #9
    My project that builds in six seconds is written in C/Objective C. It includes .c files that perform complex mathematical calculations. I couldn't see any reason to encapsulate these as objects because object management involves some overhead and I need high-performance computing. Swift does things like automatically check for array overrun. I didn't want the performance penalty. Also my iMac has 16GB and the project is running on an SSD. Maybe this is why it builds so fast. The SSD really helps performance for a lot of tasks. Also +1 on a big display for developers. I find the 21" display helpful because I can look at XCode and the App I'm testing. Since they made XCode a single unified App in one window, it's a screen hog.
     
  10. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #10
  11. Senor Cuete macrumors regular

    Joined:
    Nov 9, 2011
    #11
    That article and the one to which it links are really interesting. Now I'm SO glad that I never "upgraded" to Swift. On my machine, after I edit a source code file, building is more or less instantaneous. I have so much more important things to do than wait for XCode.
     
  12. Krevnik macrumors 68040

    Krevnik

    Joined:
    Sep 8, 2003
    #12
    I find this comment somewhat interesting, because it suggests your codebase is small enough that Swift won't make as much a perceptual difference during build. If your incremental builds are "more or less instantaneous", Swift is going to be 1.5x "more or less instantaneous". That's a far cry from the sort of measurements being discussed in the articles. If your builds are already in the two minute range, perhaps Swift adoption should wait unless the benefit outweighs the build time cost.

    Although, I'm glad our project isn't using extensive Swift, mostly because our build times are already longer than most people are used to. In general, I wouldn't convert existing projects to Swift, but I would consider it for new ones. I do use Swift for personal projects, because the language benefits outweigh the minor compile time cost, and even my larger projects can build + unit test in under 5 seconds using incremental builds.

    Honestly, in your case, it's the runtime performance hit of Swift over C (Obj-C doesn't really keep up with C any better once ARC is involved), and the easier time bridging Obj-C to C that would make me think twice about using Swift. That said, now I want to play with bridging C and Swift to see if it has gotten any better with 3.x or 4.x.
     
  13. Senor Cuete macrumors regular

    Joined:
    Nov 9, 2011
    #13
    I don't use ARC in the project either because it was originally written before garbage collection, for me manual memory management is easy and the code analyser finds all potential memory leaks. I keep hearing about things like "the pitfalls of C" but I don't know what this means. It seems like new languages and frameworks are always being invented to make programming a no-brainer. HA! programming will always be a brainer. It's like trying to make something idiot proof. Good luck with that.
     
  14. Krevnik macrumors 68040

    Krevnik

    Joined:
    Sep 8, 2003
    #14
    Eh, for me its more about where I place my effort. Writing low level, dealing with tight algorithms, sure. Manage your memory. But my smaller projects tend to be things where I'd prefer to expend my energy more specifically on functionality and overall perf than micromanage my allocations and refcounts by hand. And a lot of these projects are more likely to benefit and hit my perf targets by going async when I have to (say, generating thumbnails from video files), rather than by tightly optimizing the code behavior itself.

    For me, the simplest thing Swift does, but makes a huge impact on the quality of the code is separating the concept of a null reference away from the pointer. Being able to express in a function's contract if a parameter is really optional or not, saves me from a lot of mindless boilerplate checking nulls and the like. I still write more of that than I like, but at least it sticks around at the border between where things can be null, and where they shouldn't be allowed to ever be null.

    I don't really believe it is about making coding a no-brainer, but reducing how often I need to babysit the compiler and runtime on mindless, tedious boilerplate. And giving me the tools to attack some of this class of problem more easily prior to compile time, let alone run time.
     
  15. Senor Cuete macrumors regular

    Joined:
    Nov 9, 2011
    #15
    "...micromanage my allocations and refcounts by hand." No, you never ever do this. It's just knowing when to retain and release objects and it's quite easy if you understand it.

    "...mindless, tedious boilerplate..." Writing good code is a matter of good practices and experience. You should be able to do it. It's the opposite of mindless or tedious. It's just good programming.
     
  16. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #16
    While I agree with your first para (retain/release is not terribly hard), I disagree with the second. Removing a ton of code (retains & releases!) and immediately making related bugs not exist is wonderful.!
     
  17. robvas macrumors 68020

    Joined:
    Mar 29, 2009
    Location:
    USA
    #17
    What disk does the iMac have? That's probably the difference if it's not the 8GB of memory.
     
  18. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #18
    It's an SSD; whatever came configured by Apple.
     
  19. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #19
    Added a 2017 MBP 13":

    MacBook early 2016, 1.2GHz m5 8GB RAM (500GB SSD) Time: 4m:45s
    MBP 13" 2017 3.3GHz 16GB: 2m:33s
    MBP 15” 2015 2.2GHz i7 16GB: 2m:08s
    iMac 27” 5K late 2015 8GB 3.3GHz i5: 2m:03s

    It's far closer to the 2015 quad core that to the MacBook.

    I had ordered a 2017 MacBook but then Apple went an unleashed a bunch of refurbished 13" MBP... couldn't resist the discount.
     
  20. kage207 macrumors 6502a

    Joined:
    Jul 23, 2008
  21. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #21
    Upgraded the iMac to 16GB, putting it a little more in the lead:

    MacBook early 2016, 1.2GHz m5 8GB Time: 4m:45s
    MBP 13" 2017 3.3GHz 16GB: 2m:33s
    MBP 15” 2015 2.2GHz i7 16GB: 2m:08s
    iMac 27” 5K late 2015 8GB 3.3GHz i5: 2m:03s
    iMac 27” 5K late 2015 16GB 3.3GHz i5: 2m:00s

    All with 512GB SSD except the MBP 2015 which has 256MB.

    Rather disappointed that the iMac didn't massively beat the laptop. Or impressed the laptops are so quick.
     
  22. TokMok3 macrumors member

    TokMok3

    Joined:
    Aug 22, 2015
    #22
    I remember those times, before ARC and garbage collection. Great times indeed.
     
  23. Graham Perks thread starter macrumors member

    Joined:
    Oct 2, 2003
    Location:
    Austin, TX
    #23
    I should note that all times are with FileVault on.

    High Sierra, APFS on the MBP 13" 2017 3.3GHz 16GB: 2m:35s.

    A wash with Sierra, and means I can set aside those saying FileVault/encryption on APFS is slow. It's not materially different in performance from HFS.
     

Share This Page