Monitor URLs

[nelly22]

Now and then i notice that some Mac apps connect to net.

I try to find CLI app (or regular app), which shows what URLs these apps tries to connect. Thanks

[kryten2]

Perhaps Little Snitch can help.

Info : http://www.obdev.at/products/littlesnitch/index.html

[nelly22]

Originally Posted by kryten2

Perhaps Little Snitch can help.

Info : http://www.obdev.at/products/littlesnitch/index.html

Thanks but i need to see actual URLs.

[nelly22]

What events or notifications or ??? i need to monitor in ASOC to get all URLs visited by Safari? Thanks

[MadTester]

Originally Posted by nelly22

What events or notifications or ??? i need to monitor in ASOC to get all URLs visited by Safari? Thanks

I might be way off what you are asking but have you tried using the Developer utils in Safari? Web Inspector, Profiler? might this help? also if you expand Little Snitch it does give you a break down as to what is passing through it.

Also if you have your mac firewall on you will be able to view the log.

URL's translate to IP addresses and vice versa. Also if you look in the Utilities > 'Console' you will also see the little snitch network monitor log.

HTH...

[nelly22]

Originally Posted by MadTester

I might be way off what you are asking but have you tried using the Developer utils in Safari? Web Inspector, Profiler? might this help? also if you expand Little Snitch it does give you a break down as to what is passing through it.

Also if you have your mac firewall on you will be able to view the log.

URL's translate to IP addresses and vice versa. Also if you look in the Utilities > 'Console' you will also see the little snitch network monitor log.

HTH...

Thanks.

Little Snitch, firewall log etc don't show full URL like this: http:somedomain.com/path/index.html. I think IP cannot be translated to full URL.

I found GURL Watcher but it don't support Mountain Lion.

At this point i need only URLs which Safari visits. I wonder how it does it?

[robvas]

You could use TCP dump:

tcpdump -n -A -s1514 src 1.2.3.4 and port 80 | grep "GET\|Host:"

Replace 1.2.3.4 with the IP address of your computer

You'll get stuff like:
ost: cdn.api.twitter.com
:.....ZQGET /uds/css/small-logo.png HTTP/1.1
Host: www.google.com
:.....].GET /uds/css/v2/search_box_icon.png HTTP/1.1

[nelly22]

Originally Posted by robvas

You could use TCP dump:

tcpdump -n -A -s1514 src 1.2.3.4 and port 80 | grep "GET\|Host:"

Replace 1.2.3.4 with the IP address of your computer

You'll get stuff like:
ost: cdn.api.twitter.com
:.....ZQGET /uds/css/small-logo.png HTTP/1.1
Host: www.google.com
:.....].GET /uds/css/v2/search_box_icon.png HTTP/1.1

Thanks. That looks exactly what i'm looking for.

Unfortunately i get "tcpdump: no suitable device found" even after replacing 1.2.3.4 with my ip address from network preference pane.

Other thing is can i change it to monitor all ports, not just 80?

[MisterKeeks]

Have you tried TCPBlock

[pitaya]

Originally Posted by nelly22

Unfortunately i get "tcpdump: no suitable device found"

sudo tcpdump ...

Other thing is can i change it to monitor all ports, not just 80?

Yeah, snip off the "and port 80" part of the expression.

[jared_kipe]

TCPDump is only particularly good at getting a snapshot at a specific point in time.

The only dynamic way I now of would be to use a 'hardcore' program like wireshark to intercept all IP traffic and then parse through it looking for whatever you're after. (you'll need Xwindows or something installed, I haven't used it for a while)

Slightly easier might be using 'iftop' from the command line, but you'll need to install it through MacPorts (and have Xcode), and very high throughput will crash it (like a 60+Mbps).

You mentioned Safari. What is wrong with using the built in developer tools->Insturments->Network Requests? (see screenshot)

You can generally see every url (full url on hover). and even see them happen in psudo real time.

[nelly22]

Sorry i didn't explain it accurately in the first place. I can't use Safari developer tools, because i need text log file. Log don't need to be nice looking as long it logs.

http://www.quicomm.com/gurl_watcher_help_osx.html

"Have you tried TCPBlock"

No and it looks like it's overkill for my use and i think it don't log full URLs.

"Other thing is can i change it to monitor all ports, not just 80?
Yeah, snip off the "and port 80" part of the expression."

Cool.

"sudo tcpdump ..."

With sudo it works, but date and time stamp is still needed. And if possible, name of application which does this connection. I know nothing about grep.

I would like to also try iftop. What is easiest way to install MacPorts?

Thanks

Originally Posted by jared_kipe

TCPDump is only particularly good at getting a snapshot at a specific point in time.

The only dynamic way I now of would be to use a 'hardcore' program like wireshark to intercept all IP traffic and then parse through it looking for whatever you're after. (you'll need Xwindows or something installed, I haven't used it for a while)

Slightly easier might be using 'iftop' from the command line, but you'll need to install it through MacPorts (and have Xcode), and very high throughput will crash it (like a 60+Mbps).

You mentioned Safari. What is wrong with using the built in developer tools->Insturments->Network Requests? (see screenshot)

You can generally see every url (full url on hover). and even see them happen in psudo real time.

[web_god61]

lsof |grep TCP | grep ESTAB

[MadTester]

Posted in error

[jared_kipe]

Originally Posted by nelly22

Sorry i didn't explain it accurately in the first place. I can't use Safari developer tools, because i need text log file. Log don't need to be nice looking as long it logs.

http://www.quicomm.com/gurl_watcher_help_osx.html

"Have you tried TCPBlock"

No and it looks like it's overkill for my use and i think it don't log full URLs.

"Other thing is can i change it to monitor all ports, not just 80?
Yeah, snip off the "and port 80" part of the expression."

Cool.

"sudo tcpdump ..."

With sudo it works, but date and time stamp is still needed. And if possible, name of application which does this connection. I know nothing about grep.

I would like to also try iftop. What is easiest way to install MacPorts?

Thanks

iftop won't give you date and time text log output.

Sounds like you need to jump into the deep end with Wireshark.

[pitaya]

You could write up a script to parse tcpdump output, similar to this:
http://n3t.awardspace.us/content/tcpdump-url-extraction

It would have to be modified for OS X, and you want a timestamp:

#!/bin/bash
#
 
# reset variables
myhost="";
myurl="";
 
tcpdump -s 0 -w - -l $@ | strings |
while read line;
	do 
 
# filter GET requests
	myurl=`echo $line | grep GET | sed -E "s/GET (.*) HTTP.*/\1/"`;
	if [ "$myurl" == "" ]; then myurl=$myoldurl; fi
 
# filter Host headers
	myhost=`echo $line | grep Host | sed -E "s/Host: (.*)/\1/"`;
	if [ "$myhost" == "" ]; then myhost=$myoldhost; fi
 
# once we have a data pair, put them together and echo
	if [ "$myhost" != "" ] 
		then
		url="http://$myhost$myurl";
		echo -n "$(date): "
		echo $url;
		myhost="";
		myurl="";
	fi
 
	myoldurl=$myurl;
	myoldhost=$myhost;
done

Alternatively, just write out the packets to a file and analyze it later with whatever tool you want (tcpdump's -w flag, -r to read back packets from the file, ethereal/wireshark, etc).

[nelly22]

Thanks, this looks cool.

I saved your script to plain text file test_fs.sh.

Then i run this in Terminal app:

chmod +x /Users/Nelly/Desktop/test_fs.sh
sudo /Users/Nelly/Desktop/test_fs.sh

I cannot find log file anywhere. I think echo row(s) need something?? It don't have to save data after every url, just now and then.

When i cancel it, i get this:

^C577 packets captured
8060 packets received by filter
7371 packets dropped by kernel

Originally Posted by pitaya

You could write up a script to parse tcpdump output, similar to this:
http://n3t.awardspace.us/content/tcpdump-url-extraction

It would have to be modified for OS X, and you want a timestamp:

#!/bin/bash
#
 
# reset variables
myhost="";
myurl="";
 
tcpdump -s 0 -w - -l $@ | strings |
while read line;
	do 
 
# filter GET requests
	myurl=`echo $line | grep GET | sed -E "s/GET (.*) HTTP.*/\1/"`;
	if [ "$myurl" == "" ]; then myurl=$myoldurl; fi
 
# filter Host headers
	myhost=`echo $line | grep Host | sed -E "s/Host: (.*)/\1/"`;
	if [ "$myhost" == "" ]; then myhost=$myoldhost; fi
 
# once we have a data pair, put them together and echo
	if [ "$myhost" != "" ] 
		then
		url="http://$myhost$myurl";
		echo -n "$(date): "
		echo $url;
		myhost="";
		myurl="";
	fi
 
	myoldurl=$myurl;
	myoldhost=$myhost;
done

Alternatively, just write out the packets to a file and analyze it later with whatever tool you want (tcpdump's -w flag, -r to read back packets from the file, ethereal/wireshark, etc).

[pitaya]

You might want to pipe it through tee, or just append it to a log file:

sudo /Users/Nelly/Desktop/test_fs.sh | tee -a urls.log

sudo /Users/Nelly/Desktop/test_fs.sh >> urls.log

[Q-chan]

Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.

Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time.

Good luck and Happy New Year,

Manfred

[robvas]

Originally Posted by Q-chan

Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.

Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time.

Good luck and Happy New Year,

Manfred

Is snort or any other IDS tools available in homebrew?

[Q-chan]

Originally Posted by robvas

Is snort or any other IDS tools available in homebrew?

I'm not sure (and tend to doubt it). I prefer to use the more traditional approach of "configure --> make --> make install". On Mountain Lion you might need to build autoconf and automake, as they are no longer in Xcode (), but make sure NOT to replace libtool.

You can always try to run the configure script without these tools installed, the script will tell you when a tool is missing.

Manfred

[ipsychedelic]

Wireshark is what I use. Never required anything else, on OS X.
Just make sure, if you decide to use it, to get proficient (30 mins) on usage of filters, so you can filter out garbage you don't wanna "listen to".

[Q-chan]

Originally Posted by ipsychedelic

Wireshark is what I use. Never required anything else, on OS X.
Just make sure, if you decide to use it, to get proficient (30 mins) on usage of filters, so you can filter out garbage you don't wanna "listen to".

Good tool, but the OP needs text (text file ?) output, not X11 screens....

[ipsychedelic]

Originally Posted by Q-chan

Good tool, but the OP needs text (text file ?) output, not X11 screens....

You're right, I was quick on the response but didn't really bothered to read the whole topic (somebody had suggested Wireshark before anyway).

I think wireshark can be called from the command line, or one can use TShark. But my usage has always been in the GUI.

[robvas]

Originally Posted by Q-chan

I'm not sure (and tend to doubt it). I prefer to use the more traditional approach of "configure --> make --> make install". On Mountain Lion you might need to build autoconf and automake, as they are no longer in Xcode (), but make sure NOT to replace libtool.

I've played enough cat and mouse with libraries and packages over the years. I just checked brew and there's a snort formula availab.e