Do Mac modems have the ability to transmit voice? From what I understand not all modems do as they are designed to transmit data rather than voice. Yes I understand that the voice is packetized on the Mac end coming off the internet, but the voice coming off the POTS telephone would need to be packetized by the modem or the software. If not then the idea is a non-starter.
Seems it would make more sense to get an ATA device and put it on your network or attach is to the Mac via ethernet port and sharing and achieve pretty much the same thing. I know Skype sells specific Skype ATA devices and so do other VoIP providers like Vonage. If you sign up with one of the various SIP providers you may be able to use pretty much any SIP telephone on your network without an ATA or an ATA on the network programmed with the correct SIP information and then the old analog telephone in one of the ATA ports.