Apple has put in a feature where you double tap with two fingers and it zooms in. It's called "smart zoom". It seems that when this is enabled, secondary click slows down. I guess it's the system trying to figure out what you are going to do. If you don't think you need this, go to system preferences, then trackpad, and unlick "smart zoom". This should speed up your secondary click up to normal.