Are you sure the problem isn't that the OS itself has a limit on how many connections it can take? I've never used OSX as a home server so, while I'd assume it doesn't have one, I can't say for sure.
I have a home server that's a Win7 box running iTunes. At any given time, I have the following connected to it:
- my iMac (takes 1/5 "authenticated" licenses)
- eldest child's iMac (takes 2/5 "authenticated" licenses)
- middle child's PC (takes 3/5 "authenticated" licenses)
- the server itself takes 4/5 "authenticated" licenses
- Apple TV in master bedroom
- Apple TV in 2nd living (den)
- Apple TV in Primary Living
- My iPad Mini
- "Family" iPad 2
- Middle child's "iPhone 3GS" (used as an iPod Touch)
- Eldest child's iPhone 4
- Youngest child's/family iPad 1
- My iPhone 5
- Wife's iPhone 5
- Youngest Child's "iPhone 3G" (again, used as an iPod Touch)
At any given time, all of those are connecting to and streaming video from the server using home sharing. Over 2 years with this setup and I've not one time hit any sort of device limit on how many devices can log into Home Sharing, with the obvious exception of the computers themselves: they do require the system to be an iTunes Store authenticated device before they're allowed to login to Home Sharing. But other than those, I've never had any sort of device limitation, and any connectivity issues I've ever had have always been resolved by:
- Rebooting the server;
- Rebooting the device/Apple TV;
- Troubleshooting network connectivity issues.
All that to say: I'd rule out any potential "device limit" and focus on system limitations or network connectivity issues.