It's really a long process. You have to decide the overall design, of course. Graphics, generally, have not played a big part in the early stages. That would be icing on the cake.
You would need to decide on hardware and then how the bootstrap loader will work with the hardware to run itself and later, to load the O.S. and start the kernel and services. Last of all would be the user interface.
Of course, with Mac OS X, they had source code and hardware. It was a matter of matching what they had to work together. Once they did that, they could go about modifying that to run.
If you don't have an O.S. on your choice of hardware, you work with what you can find and use a cross-compiler and/or a cross-assembler. These allow you to create the binary code which actually runs on your desired hardware.
I would imagine that work on Macintosh started on the Lisa. Perhaps, work on the Lisa started on a 68000 UNIX platform that was available in the late 1970s and early 1980s.