F1 2017 - performance comparison

[imacken]

OK, so I've spent a lazy Sunday morning doing quite a few tests on the graphical performance of F1 2017 both on Windows and macOS.
Windows performance was on Bootcamp with latest Crimson 17.7.2 drivers.
See attached graphic for fps obtained from in-game benchmark test set to Australia and cockpt view.
All tests were with AA on. (Switching it off made a consistent 4-5 fps increase across all res and settings, and that isn't enough to consider keeping AA off as it does make quite a difference to graphic quality.)
Firstly, I have got to say a well done to Feral - as usual - for getting this game out on the same day as the Windows version, and for doing a great port.
My conclusion after looking at the results is:
Bad news - macOS and Metal is still some distance behind Windows and DX. One resolution down to be exact in this game. On the same resolution, Windows fps is up to 40% higher than the Mac version. Interestingly, the Mac performance is almost the same @ 1080p at all settings as the Windows version @ 1440p.
Good news - both versions look as good as each other in eye candy terms, which wasn't always the case in the past. In addition, I really can't tell much difference between 1440p and 1080p in visual terms.
Conclusion - I can get a respectable 44fps on my iMac with High settings @ 1080p, and that is good enough for me.

NOTE: Tested on late 2015 27" iMac 4GHz with M395X 4Gb

 

[jeanlain]

Given that F1 2016 runs faster on macOS than on Windows (at least on the Macs where it's been tested), I expected F1 2017 to do better...

 

[imacken]

Not sure where you got that from, so I ran the tests on 2016 as well and the results showed the exact same pattern as 2017.
Can't compare the numbers between 2016 and 2017 as it is not possible to select a view in 2016. The game cycles through the various view during the benchmark.
Sure the tests you did were using the latest AMD Crimson drivers on bootcamp and not the poor Apple ones?

 

[jeanlain]

Not sure where you got that from

https://egpu.io/forums/mac-setup/me...fps-on-macos-10-12-4-fhd-ultra-high-settings/
https://www.mac4ever.com/actu/12005...indows-avec-f1-2016-metal-fait-des-merveilles

A better comparison test between APIs should also always enable V-Sync if it cannot be disabled in macOS (that is, in all Metal games except Hitman, which uses Metal for the rendering and OpenGL for the display).
That matters even if performance is below 60 fps. V-Sync will always reduce frame rate because every frame will have to way for the next refresh event.

 

[imacken]

Yes, in these cases it is not at all clear how they have tested the Windows version. I suspect they are just using the 'normal' Windows Apple bootcamp drivers. Using the actual AMD Crimson drivers makes a huge difference and brings bootcamp into line with a native Windows install from a GPU point of view.
My results are pretty conclusive I would say. Consistent across the board. Agreed about vsync, but it's only going to make a few fps difference, not enough to affect the test results.

 

[jeanlain]

Yeah they probably used the default divers. But I don't expect them to reduce performance by half, which is what it would take to make the Windows version so much slower in F1 2016.

 

[imacken]

Not sure where this conversation is going, but what do you mean by ‘reduce performance by half’?

 

[JordanNZ]

Something is weird there. F1 2016 should run just as well on macOS as windows on AMD hardware.

Not sure about 2017... but it shouldn't be that much of a difference.

 

[imacken]

Something is weird there. F1 2016 should run just as well on macOS as windows on AMD hardware.

Why do you say that?

 

[jeanlain]

Not sure where this conversation is going, but what do you mean by ‘reduce performance by half’?

If you assume that that your test is representative of what Windows performance should be in F1 2016 with crimson drivers, then it should be 40% faster than macOS, just like you observed on F1 2017. Take for instance the mac4ever test at 1080p. Add 40% to the macOS results and you get 81.2 fps instead of the 39 fps measured.

 

[imacken]

If you assume that that your test is representative of what Windows performance should be in F1 2016 with crimson drivers, then it should be 40% faster than macOS, just like you observed on F1 2017. Take for instance the mac4ever test at 1080p. Add 40% to the macOS results and you get 81.2 fps instead of the 39 fps measured.

The mac4ever test is very suspicious, and makes no sense. With the exception of their highest resolution, there is no difference between any of their results. Basically, they are all 40fps. As I said, makes no sense.
The effect of the Crimson drivers should not be underestimated. When I did test a year or so ago on Tomb Raider and Deus Ex, I’m pretty sure they were showing about 30% improvement on Apple’s basic Bootcamp drivers.
You have to be comparing like for like, and, ideally, compare the Windows performance on a dedicated Windows PC with the exact same spec as the Mac you are testing, not on Bootcamp with bad drivers. Obviously, that is impossible.
My tests were reasonably scientific as they were conducted over several resolutions and a set of different settings - the same on both platforms. In addition, to that, the results are all consistent with each other, and were conducted on hardware that we know the exact specs of.

 

[jeanlain]

The mac4ever test is very suspicious, and makes no sense. With the exception of their highest resolution, there is no difference between any of their results. Basically, they are all 40fps. As I said, makes no sense.

On the contrary, it shows that performance is CPU-bound at some point, hence it should not depend on screen res. That's typically seen at low resolutions with a good GPU.
[doublepost=1503870460][/doublepost]

You have to be comparing like for like, and, ideally, compare the Windows performance on a dedicated Windows PC with the exact same spec as the Mac you are testing, not on Bootcamp with bad drivers. Obviously, that is impossible.

Bootcamp drivers are just older AMD drivers, which will always be more optimised than macOS drivers. GPU vendors will never spend nearly as much time on the Mac as they do for Windows, and more importantly, they will never optimise drivers for a specific macOS title (on Windows, they go as far as replacing the game engine shaders with their own at runtime if it gives them a few more fps).
Anyway, using the same Mac with non-bootcamp drivers would be a valid comparison. The aim is to compare OSes, not to compare the best performance one could achieve in both platforms regardless of the hardware.

 

[imacken]

On the contrary, it shows that performance is CPU-bound at some point, hence it should not depend on screen res. That's typically seen at low resolutions with a good GPU.

Exactly, that's why a limited system that can't perform the test correctly is pointless. If you're going to do a test, make sure it's relevant.

Anyway, using the same Mac with non-bootcamp drivers would be a valid comparison. The aim is to compare OSes, not to compare the best performance one could achieve in both platforms regardless of the hardware.

Exactly, that was/is my point!

 

[JordanNZ]

Why do you say that?

Feral Devs etc. You shouldn't have that much of a difference.

 

[jeanlain]

Exactly, that's why a limited system that can't perform the test correctly is pointless. If you're going to do a test, make sure it's relevant.

It can perform the test just as any other supported Mac. On a desktop, you get a more powerful CPU, but a better GPU as well, so I expect to see the same pattern. 1080p is the base resolution these days, and performance is expected to plateau at anything below as long as you have a descent GPU. It's actually a good mean to compared APIs since they mostly affect CPU time, not GPU time. Of course, one would not compare GPUs at low resolutions.
With that being said, F1 2016 appears to perform rather poorly on boot camp. So either their is a serious issue with drivers, or the game is poorly optimised, which is why Feral may have been able to do better than the original coders. Maybe F1 2017 on Windows is more optimised.
Feral may also have had less time for optimisation since they apparently targeted a simultaneous release with the Windows version.
Of course, it could be any other issue related to some Metal missing features or drivers issues that affect this particular title, or even an issue with your specific GPU drivers. There's something wrong if the latest and greatest macOS API (which is supposed to involve much lower CPU time than directX) and the greatest macOS game developers are beaten by 40%.
It'd be nice if Feral coud tell us about their own testing, which they sometime do here.

 

[Agent OrangeZ]

OK, so I've spent a lazy Sunday morning doing quite a few tests on the graphical performance of F1 2017 both on Windows and macOS.
Windows performance was on Bootcamp with latest Crimson 17.7.2 drivers.
See attached graphic for fps obtained from in-game benchmark test set to Australia and cockpt view.
All tests were with AA on. (Switching it off made a consistent 4-5 fps increase across all res and settings, and that isn't enough to consider keeping AA off as it does make quite a difference to graphic quality.)
Firstly, I have got to say a well done to Feral - as usual - for getting this game out on the same day as the Windows version, and for doing a great port.
My conclusion after looking at the results is:
Bad news - macOS and Metal is still some distance behind Windows and DX. One resolution down to be exact in this game. On the same resolution, Windows fps is up to 40% higher than the Mac version. Interestingly, the Mac performance is almost the same @ 1080p at all settings as the Windows version @ 1440p.
Good news - both versions look as good as each other in eye candy terms, which wasn't always the case in the past. In addition, I really can't tell much difference between 1440p and 1080p in visual terms.
Conclusion - I can get a respectable 44fps on my iMac with High settings @ 1080p, and that is good enough for me.

NOTE: Tested on late 2015 27" iMac 4GHz with M395X 4Gb

Thanks for the mention. I appreciate it.

 

[imacken]

It can perform the test just as any other supported Mac. On a desktop, you get a more powerful CPU, but a better GPU as well, so I expect to see the same pattern. 1080p is the base resolution these days, and performance is expected to plateau at anything below as long as you have a descent GPU. It's actually a good mean to compared APIs since they mostly affect CPU time, not GPU time. Of course, one would not compare GPUs at low resolutions.
With that being said, F1 2016 appears to perform rather poorly on boot camp. So either their is a serious issue with drivers, or the game is poorly optimised, which is why Feral may have been able to do better than the original coders. Maybe F1 2017 on Windows is more optimised.
Feral may also have had less time for optimisation since they apparently targeted a simultaneous release with the Windows version.
Of course, it could be any other issue related to some Metal missing features or drivers issues that affect this particular title, or even an issue with your specific GPU drivers. There's something wrong if the latest and greatest macOS API (which is supposed to involve much lower CPU time than directX) and the greatest macOS game developers are beaten by 40%.
It'd be nice if Feral coud tell us about their own testing, which they sometime do here.

I disagree with you about that test. It is pointless testing performance of anything if the test kit can’t give representative results. But let’s leave that one there.
Regarding F1 2016, it performs better in Bootcamp with proper drivers than F1 2017 does as you can see from the results I posted earlier.
As you said, it would be nice to have Feral’s comments.

 

[jeanlain]

The test reports the average fps over a long period of time. Frame rate being capped at 60 can really penalise the Mac, if the game renders more than 60 fps during significant parts of the test (but only 60 fps can be displayed). FPS may have hit 60 even if the game reports much less in average. Have you checked with an fps counter?
Again, I would test with V-Sync on on Windows (or generate fps graphs on both OSes).

But that wouldn't explain why the other tests found macOS to be faster, since they tested with V-Sync off on boot camp (a least the first link I posted did, I'm not sure about the mac4ever test since frame rate never exceeds 60 on Windows).

Also, the Macbook Pro mac4ever tested achieved 47 fps at 2304*1440 and 41 fps at 2560*1600. We may extrapolate ~45 fps at 1440p, which is what your iMac achieved at medium settings on F1 2016.
The M395X is rated at 3.7 GFlops while the radeon Pro 460 can only do 1.86 GFlops.
Maybe they used low settings (they didn't say). Can you run the test on low? Or perhaps performance heavily depends on the track used (I don't know if you can test performance on different tracks in this game).
There's a possibility that something is wrong with the M395X, or with you particular config, on this game engine.

EDIT: note also that in the first link I posted, they tested a gtx 980, which performed as expected relative to the RX 480 (i.e., only slightly faster). The Maxwell card cannot use boot camp drivers I guess (since there's no Mac shipping with such card). So there's no evidence that the RX 480 performance is completely ruined by boot camp drivers. It's even possible that they used regular AMD drivers (wouldn't be surprising from these eGPU geeks).

 

[imacken]

You’ve hit the nail on the head. It is impossible for us to know how the 2 tests you mentioned were carried out. We don’t know what settings were used, etc. They look pretty random to me.
I have no idea why you would hold these tests in any esteem compared to my reasonably scientific ones. Are you in denial? I really can’t understand your attitude here. There is absolutely nothing wrong with my configuration, and it shows better FPS performance given the hardware than your linked tests, particularly in Windows.
From what you say, I don’t think you even own these games, so how can you comment with any authority?
Why are you questioning my results?
I posted the test results as a useful guide for people, so I hope others can take them for what they are, i.e. a demonstration that, given the proper drivers, that the Mac and Metal performance still lags behind Windows and DX. I don’t think that will come as a shock to many people.
The good news is that we wouldn’t have these (and others) recent games at all on Mac if it wasn’t for Metal, so for that, we should be thankful.
Well done Feral, keep up the great work!

 

[marksatt]

You’ve hit the nail on the head. It is impossible for us to know how the 2 tests you mentioned were carried out. We don’t know what settings were used, etc. They look pretty random to me.
I have no idea why you would hold these tests in any esteem compared to my reasonably scientific ones. Are you in denial? I really can’t understand your attitude here. There is absolutely nothing wrong with my configuration, and it shows better FPS performance given the hardware than your linked tests, particularly in Windows.
From what you say, I don’t think you even own these games, so how can you comment with any authority?
Why are you questioning my results?
I posted the test results as a useful guide for people, so I hope others can take them for what they are, i.e. a demonstration that, given the proper drivers, that the Mac and Metal performance still lags behind Windows and DX. I don’t think that will come as a shock to many people.
The good news is that we wouldn’t have these (and others) recent games at all on Mac if it wasn’t for Metal, so for that, we should be thankful.
Well done Feral, keep up the great work!

I (only occasionally, admittedly) run UE4 performance tests on my Mac Pro in basically the same way you have tested these games sir. I like my comparisons to be Apple's to apple's so to speak and Mac's are routinely kitted out with rarer or semi-custom GPUs so it isn't practical to find a truly equivalent Windows box.

That AMD don't supply up-to-date drivers for their Mac GPUs when running Windows is a problem, so running the community patched drivers is entirely fair as there *will* be more optimisations in the newer drivers. There's not much point comparing to where D3D was a year or two ago.

Metal on Mac is limited to 120Hz by CoreAnimation, not 60Hz, so that shouldn't affect the numbers until you start hitting that cap. Doesn't look like you will here.

There's quite a bit of variation in AMD's GPU designs - the marketing names don't necessarily tell the whole tale - the M395X is a newer design than the rest of the M3*** lineup. So to really know how a game is performing means testing on a wide range of hardware.
[doublepost=1504055558][/doublepost]One thing to note is that you should manually change the screen resolution in System Preferences to the non-Retina resolution you want to play at (e.g. 1920x1080, 2560x1440 etc) unless Feral have the game switching for you automatically. This is because by default macOS will have to scale the game output to the full 5120x2880 which eats *significant* GPU time. Windows games rarely do this - they will actually switch the display resolution and so are pushing fewer pixels.

 

[jeanlain]

I have no idea why you would hold these tests in any esteem compared to my reasonably scientific ones. Are you in denial? I really can’t understand your attitude here. There is absolutely nothing wrong with my configuration, and it shows better FPS performance given the hardware than your linked tests, particularly in Windows.
From what you say, I don’t think you even own these games, so how can you comment with any authority?
Why are you questioning my results?

I'm sorry that you though I questioned your results. That's not the case. I don't doubt that anyone else with the same Mac as yours will get the results you obtained. I also don't doubt that any user with a Macbook Pro and a Polaris GPU will get the results the other guys obtained. Yes, it's unfortunate that they didn't disclose the game settings, but that shouldn't matter as long as these were the same across OSes. A few things suggest these tests are valid:
- Results are roughly similar in both tests (i.e., macOS is faster), which makes a testing error unlikely.
- Results from the GTX 980 exclude a major issue with boot camp drivers affecting the radeon results on Windows.
Also, Brad Olivier, who ported these games, came across these results and said:

you noted our F1 2016 mac vs. Windows Boot Camp perf.

If Windows was faster than macOS in his own tests, he wouldn't have said that. Yes, he said "Boot Camp" specifically, but see my second point above.

All that doesn't explain the difference with your results. I'm totally fine with macOS being beaten by Windows on you Mac. I just want to understand why. The other tests suggest it's not Feral's fault, but possibly an issue with some particular Metal GPU drivers that still need more optimisation.
[doublepost=1504076028][/doublepost]

Metal on Mac is limited to 120Hz by CoreAnimation, not 60Hz, so that shouldn't affect the numbers until you start hitting that cap. Doesn't look like you will here.

FI 2016 is capped at 60 fps apparently. Measured frame rates can't go above that in either the internal benchmark tool or in CountIT (which does measure up to 120 FPS in other Metal games like War Thunder).
[doublepost=1504076297][/doublepost]

This is because by default macOS will have to scale the game output to the full 5120x2880 which eats *significant* GPU time.

I tested some games in windowed mode (War Thunder doesn't upscale in that mode, which makes the window very tiny on my 5K iMac) and in fullscreen mode. I never measured a difference between modes. I may try in non-5K fullscreen mode.

As I side note, I like that it's the Window server, and not the display hardware, that does the upscaling, as it neatly quadruples the pixels if you run the game at a quarter of the native monitor resolution (e.g., 1440p on 5K). The display hardware (at least on almost all monitors) will do some interpolation that blurs everything at any non-native res. Windows gamers have been complaining about it for ages. The macOS approach also makes shifting between games and other apps much more pleasant.

 

[imacken]

All that doesn't explain the difference with your results. I'm totally fine with macOS being beaten by Windows on you Mac. I just want to understand why. The other tests suggest it's not Feral's fault, but possibly an issue with some particular Metal GPU drivers that still need more optimisation.

I don't think anyone is saying this is Feral's fault. Things are what they are.

FI 2016 is capped at 60 fps apparently.

Both F 2016 and 2017 are capped as vsync cannot be switched off.

The bottom line is that the tests you mentioned are in no way useful in my opinion, as they are either limited by hardware or graphic drivers. We have no idea how they did their testing at all, and in one of them there is a cursory reference to Windows results '~55fps'.
I am going to install Hitman on Windows and test that, so I will post the results later.

 

[Janichsan]

I am going to install Hitman on Windows and test that, so I will post the results later.

I can help with that:

macOS:

Benchmark Results:
---- CPU ----
4521 frames
 39.38fps Average
 10.09fps Min
139.96fps Max
 25.39ms Average
  7.14ms Min
 99.15ms Max

Windows:

Benchmark Results:
---- CPU ----
5388 frames
 46.36fps Average
  1.15fps Min
215.12fps Max
 21.57ms Average
  4.65ms Min
867.15ms Max

Except for the "Mirror quality" (which cannot be changed in the Options menu) the settings are the same on both sides.

Graphics Settings:
 RESOLUTION: 1920 x 1200
 ResolutionWidth = 1920
 ResolutionHeight = 1200
 Refreshrate = 0
 Fullscreen = 1
 ExclusiveFullscreen = 0
 VSync = 1
 VSyncInterval = 1
 Monitor = 0
 Adapter = 0
 Aspectratio = 0
 WindowPosX = 480
 WindowPosY = 300
 WindowWidth = 1920
 WindowHeight = 1200
 Stereoscopic = 0
 Stereo_Depth = 3.000000
 Stereo_Strength = 0.030000
 WindowMaximized = 0
 FocusLoss = 0
 UseGdiCursor = 0
 HDR = 0
 ShadowQuality = 3
 ShadowResolution = 1
 TextureResolution = 1
 TextureFilter = 4
 SSAO = 0
 MirrorQuality = 1 (0 on macOS)
 AntiAliasing = 1
 LevelOfDetail = 2
 MotionBlur = 0
 Bokeh = 0
 SuperSampling = 1.000000
 Gamma = 1.000000
 QualityProfile = 4

Note that VSync was enabled, but the game seems to ignore that for the benchmark.

 

[jeanlain]

I did a lot of tests with Hitman, with fps graphs and all. Performance on macOS is basically between 80% and 100% that of DX11 (boot camp), depending on the resolution.

 

[imacken]

I can help with that:

Sure you are talking about Hitman, and not Hitman: Absolution? I don't see half of these options, and I also can't see how to do a benchmark test!