Of course you can create custom fonts in MacOS X. FontLab's
FontLab Studio, its veteran font editor
Fontographer, and several other excellent font creation tools can be found
here.
There are a lot of really bad fonts created by experienced amateurs. The novice does not stand a chance. If you need a custom font, then you will be much better served by hiring a professional to do the job for you. If you can't afford a professional type designer, then select a commercial font that is close to what you want. You can convert your characters to outlines and tweak them to a closer approximation of what you envision.