Go Back   MacRumors Forums > Apple Systems and Services > OS X > Mac OS X Server, Xserve, and Networking

Reply
 
Thread Tools Search this Thread Display Modes
Old Jul 23, 2010, 04:18 PM   #1
darkplanets
macrumors 6502a
 
Join Date: Nov 2009
[Solved] UDP Firewall Conundrum

This should probably go in the networking section, but alas, responses and uptake are slow in that part of the forum, so I think this is preferable. This can also help other members.

The problem is as the title suggests; the UDP ports with my software based firewall. More specifically, I'm using VirusBarrier X6, but this shouldn't really matter, as the firewall setup is more or less universal among competitors. Furthermore, I'm using a custom set-up, not one of the pre-configured jokes included in the program. My network path is as follows: Internet -> router -> wireless -> Mac/Software firewall -> Internet sharing -> LAN -> XBox. I use my wireless connection both for my Internet connection as well as the Xbox's connection, which will be referenced later. Also, Microsoft support doesn't provide any help.

To start off, I guess I'll post my current firewall set-up.

NOTE: This is merely one part of my firewall, simplified. When I typically make new firewall rules, I work on one section at a time, in order to iron out any inconsistencies or problems present in my port-forwarding. In order to do so, typically the other areas (in this case LAN<->Mac and Mac -> Internet) are set to be entirely open, so I can check for issues and isolate them more easily. You will see this below in my setup.

Rules (Higher on the list means earlier preference):

Internet -> Mac.
Ports
....
Other ports
....
Xbox Live In:
TCP 80, UDP 88, UDP/TCP 3074,UDP/TCP 53 (Broadcast packets allowed, destination ports)
Note that TCP 80 (http) and UDP/TCP 53 (domain) really aren't needed at all; I included them because they're recommended by Microsoft and I was having issues, so it was a troubleshooting point.
....
VPN In (LT2P over IPsec):
UDP 500, UDP/TCP 1701 (Broadcast packets allowed, destination ports)
Note that these were also provided by my VPN provider, more on this later.
....
TCP In
All ports STOP (no broadcast packets)
....
UDP In
All ports STOP (no broadcast packets, destination port)
....
Internet In
All ports GO (ICMP, IGMP)

Internet Out (Mac -> Internet)
All go.

LAN In/Out
All go.


Now that we've got that done, its time to discuss the problem(s). There are two, specifically, both revolving around the UDP blockage near the bottom of the Internet inbound list; XBox, and VPN. Both function fine without the firewall, and both function perfectly when the UDP blockage is the only thing turned off, leaving my troubleshooting pointing to a missing UDP port-exception for both services. TCP has remained blocked throughout, as noted on the list, which is located near the bottom. There have been no issues with this.

For my Xbox issues, it goes like this:

When UDP blockage and the XBox in-ports are enabled, I get an MTU error; Microsoft says says you need an MTU of 1364, and mines set on auto and is at 1500. Clearly above their requirements, yet it doesn't work.

When I turn off the UDP blockage, but keep the XBox ports going, I can connect to Live without a problem.

When I turn off the UDP blockage as well as the XBox ports, I get the same result; a perfect connection. This would leave me to believe that the XBox ports aren't so necessary in the first place, at least the ones they provided, applied to the inbound traffic, specifically in relation to TCP, as that's been blocked the entire time (outbound they're needed).

When I turn on the UDP blockage, but turn OFF the XBox ports, I get a DNS error; this is probably due to the UDP 53 port being blocked. Interestingly enough though, TCP has been blocked the entire time, meaning TCP 53 is useless. Turning off the TCP blockage on top of the aforementioned settings results in the same error. My DNS server for my xbox is my computer, listed as 192.168.2.1, as you would expect in an Internet sharing situation.

Thus, at least for my Xbox problems, I've come to the conclusion that I'm missing some UDP port in my port forwarding scheme, the only question is which one? Having UDP 53 and the others recommended by MS turned on for the inbound gives me the MTU error; I'm lost as to what I need to open to fix this, but its clearly UDP.


For my VPN issues; well, its far simpler. Using the in-bound allowed ports as listed above, my VPN works fine, provided that the UDP Blockage is turned off. If it is turned on, a get a message that I cannot connect to the L2TP server, and the VPN connection is never established. Once again, this leads me to believe I missed a port somewhere, yet I port forwarded the ones recommended by my VPN provider. Therefore I'm at a loss here too; any recommendations on getting this to work with the UDP blockage enabled?

Thanks!

EDIT: SOLVED!!
(Solution is in my last post)

Last edited by darkplanets; Jul 28, 2010 at 05:18 PM. Reason: Solved
darkplanets is offline   0 Reply With Quote
Old Jul 25, 2010, 10:02 AM   #2
satcomer
macrumors 601
 
satcomer's Avatar
 
Join Date: Feb 2008
Location: Upstate NYS
Well first thing you should do is bookmark the Apple document "Well known" TCP and UDP ports used by Apple software products is you want to really know what ports OS X uses. Then you should drop VirusBarrier's firewall and use the internal Unix ipfw. The build in ipfw has been ungraded ever since Unix was made. The following programs will allow a Mac user to use ipfw with using the command line.As a novice you could use the free NoobProof and if you are an expert then use the free WaterRoof.
__________________
Mac Pro Dual 2.8 Quad(Rev B.), 16 G RAM, OS X 10.9, 23'' LCD
Mac Book Pro Core 2 Duo 2.16Ghz, SuperDrive, 2G RAM, OS X 10.7.5
iPad 3, 32 black
satcomer is offline   0 Reply With Quote
Old Jul 25, 2010, 11:13 AM   #3
darkplanets
Thread Starter
macrumors 6502a
 
Join Date: Nov 2009
Quote:
Originally Posted by satcomer View Post
Well first thing you should do is bookmark the Apple document "Well known" TCP and UDP ports used by Apple software products is you want to really know what ports OS X uses. Then you should drop VirusBarrier's firewall and use the internal Unix ipfw. The build in ipfw has been ungraded ever since Unix was made. The following programs will allow a Mac user to use ipfw with using the command line.As a novice you could use the free NoobProof and if you are an expert then use the free WaterRoof.
I appreciate the response, but the reason I use VirusBarrier is because I'm on a mixed home network, and files go between my Mac and Windows computers regularly. Hence I really use the VB scanning utilities, so it makes sense for me to also use the built-in firewall.

I was actually using IANA's list; here. I find it to be a lot more complete, especially when setting things up for both computers

I preferred not to use the ipfw initially because it was command line; I didn't want to set up routing tables, etc, and OSX's built in GUI for a "statefull" ipfw firewall I found unacceptable. Those GUI front-end solutions do sound nice though, but when it comes down to it I really see little difference. Both are firewalls, both have front-end GUI interfaces, and both are entirely customizable. If I did not need the virus scanning capabilities WaterRoof sounds nice; I'll still probably look into it just out of curiosity.

The real issue here is that I've never had to port forward for these two services before; previously my 360 was hard-wired on the network, and I didn't use a VPN. Both seem to have a similar UDP problem, which also is relatively new; before I had a very loose firewall that only blocked TCP and ICMP, not UDP or IGMP. That's changing now, which is why I'm troubleshooting these issues now.

I do, however, want to thank you for sparking a thought that lead me to solve my second VPN problem; I forgot about the IKE NAT traversal on UDP 4500. WaterRoof's site sparked that insight, through their NAT statement

I'm still stuck on the xbox issue though; I'll have to do some more digging later.

EDIT: Very few routers support specific Protocol enabling, IE Protocol 50, 51; since I'm using UDP 4500 for IKE NAT traversal I don't have to worry about this, right? Otherwise I'd have to set Protocol 50 and 51 as allowed on my router for the NAT pass through. As a hypothetical though, I have another question. Since I'm Internet sharing and becoming my own gateway, how would it work if I had some computer connecting to a VPN through me? I can't allow IP Protocol 50 or 51 through software means--at least not in VB or WaterRoof-- could I just allow UDP 4500 again to make it work?

Last edited by darkplanets; Jul 25, 2010 at 11:50 AM.
darkplanets is offline   0 Reply With Quote
Old Jul 28, 2010, 05:17 PM   #4
darkplanets
Thread Starter
macrumors 6502a
 
Join Date: Nov 2009
Update time! I figured the Xbox issue out too, which means all of my problems are solved. Woo!

I tested batches of UDP ports in groupings of 5000, 500, 100, 50, 25, and 5 until I found what I needed; not the most efficient, I know, but I had a generally good idea where it was originating from, so I got it done pretty fast (~15 minutes). Interestingly enough, the ports needed for the XBL connection were rather... random. UDP 1257-1259. If you look these up on IANA's list, you'll see that these are listed as Shockwave 2, Open Network Library, and Open Network Library Voice. As far as I know, none of these are related to XBL in any way. Furthermore, none of the TCP flavors are required; only the UDP. TCP remains blocked, as always, and had no effect upon the connectivity or relative NAT status.

I guess what I find the most enlightening about all of this was that it originated from an MTU error; the fragmented error message wasn't being sent back between my computer and xbox. Interestingly enough, ICMP is fully open on my firewall (for now), so the error message needed for ICMP should have been sent back. Since it wasn't, it would lead me to believe that they're using PMTUD, specifically over UDP. If I kept one of the ports closed, it would give me an ICMP unreachable error (remember its open ), whereas if I kept two closed, it would just state that I couldn't reach the Internet-- overall its just very odd given the current rules setup I have.

Of course when I was trying to solve this I went the logical route and contacted Microsoft first; they had no idea. In fact, the representative I talked to didn't even understand what ICMP was, let alone MTU. It would appear as if Microsoft doesn't even know that their console uses PMTUD over UDP... not a big surprise I guess. Hence why it never appeared on any documentation, both on their website and in their employee's support documents.

Thus, to anyone having issues with XBL connections, specifically with MTU errors, open UDP 1257-1259.

If you're having VPN errors, make sure to open UDP 4500 for NAT traversal if you're using a router.

I hope this helps someone in the future.
darkplanets is offline   0 Reply With Quote

Reply
MacRumors Forums > Apple Systems and Services > OS X > Mac OS X Server, Xserve, and Networking

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Similar Threads
thread Thread Starter Forum Replies Last Post
Bad Network performance - udp T1663R Mac Programming 2 Apr 5, 2014 12:03 PM
VLC UDP stream + internet jurre OS X 10.8 Mountain Lion 1 Nov 5, 2012 10:54 AM
[Verizon] What will happen to my UDP when my contract expires apb89 iPhone 6 Oct 10, 2012 04:22 PM
ipsec over udp sealucky OS X 10.8 Mountain Lion 4 Aug 7, 2012 06:58 AM
How to find an open UDP port (ShareMouse App) zonatigre Mac Basics and Help 0 Jun 7, 2012 05:18 PM

Forum Jump

All times are GMT -5. The time now is 09:41 AM.

Mac Rumors | Mac | iPhone | iPhone Game Reviews | iPhone Apps

Mobile Version | Fixed | Fluid | Fluid HD
Copyright 2002-2013, MacRumors.com, LLC