Not having Google tracking your every move (don't forget that Google is an advertising company, that only makes products to further that interest) is reason enough for me to avoid Chrome, but making the browser "fast" by skipping things like image color management, using lower quality text rendering and poor quality image scaling (important on retina displays) are also things to consider.
Chrome is actually the most memory-hungry browser on the market right now, it's just that they split it over so many processes, that people often don't notice it. The "Chrome is fastest" mantra that I've seen posted all around the web in the last year or so, simply isn't true. The only reason I can see where people might think that, is that they're comparing a fresh browser install with no extensions etc. to an older install of Safari or Firefox that has a big history file, lots of extensions and bookmarks set up etc. If they compared a fresh install of each, they would find that it's definitely not the case.
I've also found Chrome to be far less stable than other browsers ever since they integrated Flash. (frankly, I want to avoid Flash as much as possible)
Their JavaScript performance is quick, but that's about all they have going for them now. I'd much rather use Firefox, which has the most advanced image color management solution, best text rendering and image scaling, a low memory footprint, the best extension ecosystem, and is among the fastest browsers out there. Failing that, I'd rather use Safari if I wanted a WebKit-based browser, and wanted to sync bookmarks & tabs with an iOS device or across machines. (Firefox also has sync, but no good iOS integration yet)
http://www.tomshardware.com/reviews/macbook-air-chrome-16-firefox-9-benchmark,3108-18.html