Both controller types are possible to use in OS X. The PS 3 one simply connects via Bluetooth and can be used in many games straight away. Xbox controllers need a driver to properly work in the majority of games but they're readily available and easy to configure. For ease of use and compatibility I would suggest getting a genuine Microsoft 360 unit as many 3rd party models don't work properly.
Games for Mac are, for the most part, available in all the same formats as their Windows counterparts. You just need to be sure that the game you want is available for play on OS X. For example, Batman: Arkham Asylum and Arkham City can be played on OS X but Arkham Origins cannot. Fortunately, GOG and Steam (the two most popular digital distribution services) make it very clear which games can be played on OS X.