That's not true. Sure you can learn the language quickly, but languages are much much more than just words. In fact it would take an American moving to England quite a while to learn the new* english.
I have a friend who is from Mexico who taught himself english from watching American films. He's lived here for two years, and already spoke perfect english upon arriving here, but he still complains about not grasping connotations, innuendoes, implied ideas, etc.
I think you have an overly broad definition of what it means to speak a language. By your definition, most Americans don't speak US English very well, either. (Which they don't, but that's another matter.)
Your Mexican friend issue isn't that he doesn't speak English well, it's that he doesn't have the same cultural point of reference. If you take individuals from varying socioeconomic, cultural and regional circles and put them together, you would have just as many missed connections. People who are in the same social circle tend to speak using an incredible amount of jargon. Much more so than you as an individual might think. There's also an attribution bias. If you don't get something your friend says, you pass it off as not "getting it" or any number of reasonable causes. When your Mexican friend misses the meaning of something, it's automatically because he doesn't understand the language.
I work in diverse, ad-hoc international teams. Depending on the country, a team defaults to either English or French as the internal working language. Usually English. A lot of my colleagues who learned English as a second or third language actually speak more grammatically correct English than the average American. Idiomatically correct in many cases as well. Most of them would miss half the jokes in an episode of 30 Rock. Are these people who can write papers, reports, etc at a much more proficient level than Joe the Plumber NOT fluent English speakers? Bzzzt. (Oh, relating back to the original question, this is why I believe Canadians are more culturally similar to Americans.)
What you're driving at are the differences between fluent and native and idiomatically correct. Fluent implies more-or-less grammatically correct. Native is what it sounds like. Idiomatically correct has more to do with cultural context.
I speak, read and/or write a few of these "difficult" languages with varying degrees of proficiency. What's being missed in the discussion here is that this is the level of difficulty for a monolingual native English speaker. I speak Korean at a native level. As in, if you were a Korean and I called you on the phone, there's no way you could possibly tell that I wasn't actually Korean. Japanese classes in English are much more difficult than Japanese classes in Korean. Same thing with Mandarin. Actually, travelling in China, if you know literary Korean or have a reasonable command of Kanji, it's easier to walk around drawing characters on your hand than actually learn Mandarin.
Years ago, I had to learn Arabic. NONE of it stuck. Then I was forced to learn a little Swahili. Later, I was found myself understanding more bits and pieces of Arabic. It's not strictly the same language group, just a lot of common loan words and more mutually intelligible concepts.
In this sense, I don't believe learning any one language is any more difficult than any other. People just think English is harder simply because it's so widely spoken and as a result, has such a variance. That new Indian doctor or engineer speaks, reads and writes much better English than most everyone else on the team, it's just most guys aren't going to understand the guy for a few weeks.
And no, you don't need years of immersion to develop fluency. There's a million Dutch out there who have never set foot on native English speaking soil who speak excellent, almost idiomatically correct English. I spoke French for six months when I was six years old, then twenty years later, I got dumped in a French speaking country with a few weeks of language school. It took me a few months to develop to the point where I could communicate with my guys exclusively in French without serious miscommunications. Another few to be able conduct meetings with government officials and authorities. And let me tell you, I'm a right lazy bastard and I have no more a talent for learning languages than anyone else.