Very funny you post this— that was going to be my exact recommendation, based on my similar experience on a similar setup.Heh, it is.
And I just solved it. I feel like a dumbass.
The UDP timeout for NAT on the router I was using was set to 60 seconds. Wi-Fi calling isn't very chatty and was being timed out. Once I set it to 180 seconds the problem went away.
If you use pfSense and WiFi calling has issues, look into this.