If you aren't actively using it, speech is already disabled.
Don't confuse having the functionality available (the files on the hard drive) with that feature being active. If you don't use something there is no performance hit as the process is not actively running.