Change the file suffix to ".txt" and open it in TextEdit. There's a possibility that the file is really comma-separated values, rather than an actual Excel file.
I mention this because I've seen it for a few files I got from Fidelity. The suffix was ".xls" but the actual content was CSV.
If you were running Mountain Lion, then Preview could show you .xls and .xlsx files.
Not knowing your brokerage firm, maybe there's an account preference to send you PDF files. You could check for that if you haven't already done so. Personally, if I didn't see an option online, I'd phone the brokerage and ask them about it (again, assuming you haven't done so).