It is that hard. Apple and Adobe, and other 3rd party developers, pay some very smart people some very good money to write a computer program that can figure out which pixels are meant to be that colour, and which pixels are not supposed to be that colour - and what colour they are supposed to be. All without the programmer ever actually seeing the scene that the photographer is trying to clean up.