You're gonna need a 7.1 receiver with HDMI in order to get the full 7.1 sound out of the PS3. Optical only does up to 5.1.
Buy yourself a decent receiver, and then start adding good speakers. Everyone has a different idea of what order to add them in, but I'd say get your right, left, and center channel speakers first. Then add a subwoofer. Then build out the other 2 or 4. Or, just blow a wad of cash and do it all at the same time 😀