[Solved] UDP Firewall Conundrum

[darkplanets]

This should probably go in the networking section, but alas, responses and uptake are slow in that part of the forum, so I think this is preferable. This can also help other members.

The problem is as the title suggests; the UDP ports with my software based firewall. More specifically, I'm using VirusBarrier X6, but this shouldn't really matter, as the firewall setup is more or less universal among competitors. Furthermore, I'm using a custom set-up, not one of the pre-configured jokes included in the program. My network path is as follows: Internet -> router -> wireless -> Mac/Software firewall -> Internet sharing -> LAN -> XBox. I use my wireless connection both for my Internet connection as well as the Xbox's connection, which will be referenced later. Also, Microsoft support doesn't provide any help.

To start off, I guess I'll post my current firewall set-up.

NOTE: This is merely one part of my firewall, simplified. When I typically make new firewall rules, I work on one section at a time, in order to iron out any inconsistencies or problems present in my port-forwarding. In order to do so, typically the other areas (in this case LAN<->Mac and Mac -> Internet) are set to be entirely open, so I can check for issues and isolate them more easily. You will see this below in my setup.

Rules (Higher on the list means earlier preference):

Internet -> Mac.
Ports
....
Other ports
....
Xbox Live In:
TCP 80, UDP 88, UDP/TCP 3074,UDP/TCP 53 (Broadcast packets allowed, destination ports)
Note that TCP 80 (http) and UDP/TCP 53 (domain) really aren't needed at all; I included them because they're recommended by Microsoft and I was having issues, so it was a troubleshooting point.
....
VPN In (LT2P over IPsec):
UDP 500, UDP/TCP 1701 (Broadcast packets allowed, destination ports)
Note that these were also provided by my VPN provider, more on this later.
....
TCP In
All ports STOP (no broadcast packets)
....
UDP In
All ports STOP (no broadcast packets, destination port)
....
Internet In
All ports GO (ICMP, IGMP)

Internet Out (Mac -> Internet)
All go.

LAN In/Out
All go.


Now that we've got that done, its time to discuss the problem(s). There are two, specifically, both revolving around the UDP blockage near the bottom of the Internet inbound list; XBox, and VPN. Both function fine without the firewall, and both function perfectly when the UDP blockage is the only thing turned off, leaving my troubleshooting pointing to a missing UDP port-exception for both services. TCP has remained blocked throughout, as noted on the list, which is located near the bottom. There have been no issues with this.

For my Xbox issues, it goes like this:

When UDP blockage and the XBox in-ports are enabled, I get an MTU error; Microsoft says says you need an MTU of 1364, and mines set on auto and is at 1500. Clearly above their requirements, yet it doesn't work.

When I turn off the UDP blockage, but keep the XBox ports going, I can connect to Live without a problem.

When I turn off the UDP blockage as well as the XBox ports, I get the same result; a perfect connection. This would leave me to believe that the XBox ports aren't so necessary in the first place, at least the ones they provided, applied to the inbound traffic, specifically in relation to TCP, as that's been blocked the entire time (outbound they're needed).

When I turn on the UDP blockage, but turn OFF the XBox ports, I get a DNS error; this is probably due to the UDP 53 port being blocked. Interestingly enough though, TCP has been blocked the entire time, meaning TCP 53 is useless. Turning off the TCP blockage on top of the aforementioned settings results in the same error. My DNS server for my xbox is my computer, listed as 192.168.2.1, as you would expect in an Internet sharing situation.

Thus, at least for my Xbox problems, I've come to the conclusion that I'm missing some UDP port in my port forwarding scheme, the only question is which one? Having UDP 53 and the others recommended by MS turned on for the inbound gives me the MTU error; I'm lost as to what I need to open to fix this, but its clearly UDP.


For my VPN issues; well, its far simpler. Using the in-bound allowed ports as listed above, my VPN works fine, provided that the UDP Blockage is turned off. If it is turned on, a get a message that I cannot connect to the L2TP server, and the VPN connection is never established. Once again, this leads me to believe I missed a port somewhere, yet I port forwarded the ones recommended by my VPN provider. Therefore I'm at a loss here too; any recommendations on getting this to work with the UDP blockage enabled?

Thanks!

EDIT: SOLVED!!
(Solution is in my last post)

 

[satcomer]

Well first thing you should do is bookmark the Apple document "Well known" TCP and UDP ports used by Apple software products is you want to really know what ports OS X uses. Then you should drop VirusBarrier's firewall and use the internal Unix ipfw. The build in ipfw has been ungraded ever since Unix was made. The following programs will allow a Mac user to use ipfw with using the command line.As a novice you could use the free NoobProof and if you are an expert then use the free WaterRoof.

 

[darkplanets]

Well first thing you should do is bookmark the Apple document "Well known" TCP and UDP ports used by Apple software products is you want to really know what ports OS X uses. Then you should drop VirusBarrier's firewall and use the internal Unix ipfw. The build in ipfw has been ungraded ever since Unix was made. The following programs will allow a Mac user to use ipfw with using the command line.As a novice you could use the free NoobProof and if you are an expert then use the free WaterRoof.

I appreciate the response, but the reason I use VirusBarrier is because I'm on a mixed home network, and files go between my Mac and Windows computers regularly. Hence I really use the VB scanning utilities, so it makes sense for me to also use the built-in firewall.

I was actually using IANA's list; here. I find it to be a lot more complete, especially when setting things up for both computers

I preferred not to use the ipfw initially because it was command line; I didn't want to set up routing tables, etc, and OSX's built in GUI for a "statefull" ipfw firewall I found unacceptable. Those GUI front-end solutions do sound nice though, but when it comes down to it I really see little difference. Both are firewalls, both have front-end GUI interfaces, and both are entirely customizable. If I did not need the virus scanning capabilities WaterRoof sounds nice; I'll still probably look into it just out of curiosity.

The real issue here is that I've never had to port forward for these two services before; previously my 360 was hard-wired on the network, and I didn't use a VPN. Both seem to have a similar UDP problem, which also is relatively new; before I had a very loose firewall that only blocked TCP and ICMP, not UDP or IGMP. That's changing now, which is why I'm troubleshooting these issues now.

I do, however, want to thank you for sparking a thought that lead me to solve my second VPN problem; I forgot about the IKE NAT traversal on UDP 4500. WaterRoof's site sparked that insight, through their NAT statement

I'm still stuck on the xbox issue though; I'll have to do some more digging later.

EDIT: Very few routers support specific Protocol enabling, IE Protocol 50, 51; since I'm using UDP 4500 for IKE NAT traversal I don't have to worry about this, right? Otherwise I'd have to set Protocol 50 and 51 as allowed on my router for the NAT pass through. As a hypothetical though, I have another question. Since I'm Internet sharing and becoming my own gateway, how would it work if I had some computer connecting to a VPN through me? I can't allow IP Protocol 50 or 51 through software means--at least not in VB or WaterRoof-- could I just allow UDP 4500 again to make it work?

 

[darkplanets]

Update time! I figured the Xbox issue out too, which means all of my problems are solved. Woo!

I tested batches of UDP ports in groupings of 5000, 500, 100, 50, 25, and 5 until I found what I needed; not the most efficient, I know, but I had a generally good idea where it was originating from, so I got it done pretty fast (~15 minutes). Interestingly enough, the ports needed for the XBL connection were rather... random. UDP 1257-1259. If you look these up on IANA's list, you'll see that these are listed as Shockwave 2, Open Network Library, and Open Network Library Voice. As far as I know, none of these are related to XBL in any way. Furthermore, none of the TCP flavors are required; only the UDP. TCP remains blocked, as always, and had no effect upon the connectivity or relative NAT status.

I guess what I find the most enlightening about all of this was that it originated from an MTU error; the fragmented error message wasn't being sent back between my computer and xbox. Interestingly enough, ICMP is fully open on my firewall (for now), so the error message needed for ICMP should have been sent back. Since it wasn't, it would lead me to believe that they're using PMTUD, specifically over UDP. If I kept one of the ports closed, it would give me an ICMP unreachable error (remember its open ), whereas if I kept two closed, it would just state that I couldn't reach the Internet-- overall its just very odd given the current rules setup I have.

Of course when I was trying to solve this I went the logical route and contacted Microsoft first; they had no idea. In fact, the representative I talked to didn't even understand what ICMP was, let alone MTU. It would appear as if Microsoft doesn't even know that their console uses PMTUD over UDP... not a big surprise I guess. Hence why it never appeared on any documentation, both on their website and in their employee's support documents.

Thus, to anyone having issues with XBL connections, specifically with MTU errors, open UDP 1257-1259.

If you're having VPN errors, make sure to open UDP 4500 for NAT traversal if you're using a router.

I hope this helps someone in the future.