I hear this a lot and I believe it to be incorrect. When I watch BBC shows they are often using miles, pints, stone, pounds, and gallons, so I don't think it's the ENTIRE WORLD except the USA. At the very least the UK uses it too.
What's really confusing is the car stuff, where cars are rated in miles per gallon yet petrol seems to be sold in liters (or "litres" I guess).
The Imperial system is clearly worse than the metric system, but I'd argue that using mixed systems is even worse than using the Imperial system alone!