Yes it can play H264!
Hmm, I don't know why people keep saying the original XBOX can't play H.264 files! Every one of the DVDs that I've ever encoded (yes I own them all, not pirating) has worked flawlessly. I use Handbrake on a Mac (and somtimes Linux) to encode them with H264 in an MP4 container using a target bitrate between 700-1000. I stay pretty close to their original pixel height and width. Usually the files end up being around 700-850 megs. I'm playing them back with the latest XBMC, on a Sony 480i 27" with the standard RCA connector. I'm currently shopping for a high-def TV ;-)
I can only guess that the people who are having problems with playback are trying to encode higher bitrates or play higher resolutions/upconverting using the composite HD cables. I've never tried playing higher than standard definition. My XBOX was also purchased about 2 years or so after they first came out, so I don't know if the earlier versions had lesser hardware in them.
Anyway, all I know, is that H264 works flawlessly for me!