I had this problem too on my Late 2015 5K iMac. Only the first port recognized USB 3.0 devices and ran just with USB 2.0 speed. The other ports only detected USB 2.0 devices and USB 3.0 devices connected with 2.0 cables.
But I had this only in every beta of El Capitan, not any final version and it was fixed since the 10.11.5 betas or maybe before.
What version do you have installed? All final versions had no problem from 10.11.0 to 10.11.4, just all the betas in between maybe the late 10.11.4 not anymore. I never tried it there because I mostly use Thunderbolt and did not use the iMac for a while. Started using it again in 10.11.5 Beta 3 or 4 without a USB 3.0 problem until now. At least all ports work. But the write speed was a little low as I last tested it. Read speed seemed normal.
Edit: I read only your first post. So the speed is normal now and you have other problems? I never used any USB device longer so I can't say if I have the same problems.
[doublepost=1463838569][/doublepost]I just tested it with a Sandisk Extreme 64 GB USB stick. At first it was for about five minutes only about 10/20 then about 115/240 then again very slow without stopping the test. Really strange. At the moment it's like that:
This device normally gets up to 245 MB/s.
[doublepost=1463839010][/doublepost]I changed the ports now and used the one next to Thunderbolt, what always at least detected USB 3.0 devices but had also just USB 2.0 speed. It's even worse or the same. It's starting with almost zero read speed like the other port. Maybe it also had changed after a while, but in the beginning after a few test it looks in average like that:
Edit: This is really the worst version ever. I tested it once with beta 4 of 10.11.5 and the read speed was from the beginning 245 MB/s (or even more than SanDisk states as maximum), write I can't remember but I think much less than 100 MB/s. I don't know what the normal write speed should be, but if I remember correctly it was about 200 MB/s or even the same as read.