I was one of the responsive early adopters back in early 2012. Now a year later I've just about completely purged my theme development core of responsive design ideas.
As mentioned above, I've also found that users complain of the design changes between online and mobile. Analytics from several sites show that mobile users of the responsive site have a significantly faster drop rate, backing up the user comments.
I do keep a few of the responsive ideas; I bump the font up one and 'thin it out' to make it more readable on a mobile device for example. But mostly develop fluid sites based with max/min widths set to try to keep the experience the same across all devices. I use RESS principles regarding images for differing clients, but try to limit the server calls per page to a bare minimum.
But, hey that's just me.