Actually it does. Speed of information flow has been ruled by many courts as much of a right of Free Speech as the content.
Case history goes all the way back to newspapers in the 19th century being kept from printing articles 'til after elections. The government was making claims that speech was no suppressed since they eventually distributed their content. However, the newspapers were claiming it was as they were not able to effect elections. Same goes here.
Either way, there is always a bypass. Several start-ups out here are looking to bypass IP addresses and name servers all together.
You can never get the genie back in the bottle.
Net Neutrality is a regulation that ensures speed of information flow, not the opposite. It's more like a rule that forces cops to allow protestors to protest. It's not infringing on the freedom of speech.