Whilst you can, technically, use xpostfacto to run up to 10.2 on a 603e, you really don't want to, it's horrible even on a 604e. Even if you could find one of the L2 slot G3 cards you still would have the 40 mhz bus and 136 meg RAM ceiling to deal with and that would be crippling for OS X. The OS would (possibly) run but you couldn't do anything constructive on it.
Like Hrududu my 6400/180 - Rage 128, is even pretty sluggish under OS 9 so I'd recommend 8.6 for the best performance/features balance, max out the RAM, make sure you have a L2 cache stick and bung in a cheap Rage 128 GL/Pro (a Radeon is overkill without a G3 upgrade in this machine). Whatever you do, do not try and get anything like decent performance from the onboard Valkyrie graphics chipset because it's pretty terrible.