Well, kinda. They are much slower on a per core basis (a a very rough rule of thumb 1/10 of the computing power of a CPU). But - there is a whole lot of cores, and bulk memory access is usually much faster (again; rough rule of thumb: 5x).
So, GPUs are very good if a task is parallelizable and the amount of chunks of work is huge.
Another heuristic:
CPUs - low latency, small throughput. Metaphor: Motorcycle. Propells you very fast from point A to point B. But it can can carry 2 people only, one if which is occupied by the driver, thus limiting payload
GPUs - high latency, huge throughput. Metaphor: Bus. Its pretty slow on its way from point A to point B but you can bring all your buddies. Plus your football team. Including fans
You‘d ride the motorbike if you are going alone. But if you want to bring all your friends, your football team and all your fans, then by all means the bike wouldn‘t bring you anywhere.