I don't use it but it doesn't bother me either. There is probably a way to switch it off but I don't bother as I don't have to look at it if I don't want to. I think this is OS X attempt to provide something like MS "libraries" for users that are used to seeing things that way. For those of us who have moved on from Windows years ago, it is safe to simply ignore "All my Files." In fact, I would say it's safer to ignore it than to start looking for ways to delete the thing.
One reason it doesn't bother me so much is I use the TotalFinder plugin which gives me 1 tabbed finder window rather than 20 of them along with a properly implemented "cut" feature (which I rarely need to use).