yes, but the problem is different (it won't really attempt to use a "wrong" driver, that stuff is controlled via pciids and so on), xorg doesn't like when it sees two GPUs (even if the other has a blacklisted module, it will still appear in lspci), and you need to help it decide which one to use. That's why I created a file /etc/X11/xorg.conf.d/21-gpu.conf, with contents like this
https://clbin.com/FSXZ0
The BusID numbers need to be adjusted per-machine, e.g. for my lspci listing:
https://clbin.com/LZHEO , 0001:06:00.0 becomes PCI:6@1:0:0, 0000:0a:00.0 becomes PCI:a@0:0:0.
You're free to use another xorg driver, of course. I use the modesetting default, using another DDX (ati, nouveau) may give you say, legacy-style 2D accel paths, I never bothered myself.