Register FAQ / Rules Forum Spy Search Today's Posts Mark Forums Read
Go Back   MacRumors Forums > Apple Systems and Services > Programming > Mac Programming

Reply
 
Thread Tools Search this Thread Display Modes
Old Dec 9, 2012, 08:06 AM   #1
nelly22
macrumors regular
 
Join Date: Sep 2009
Monitor URLs

Now and then i notice that some Mac apps connect to net.

I try to find CLI app (or regular app), which shows what URLs these apps tries to connect. Thanks
nelly22 is offline   0 Reply With Quote
Old Dec 9, 2012, 04:19 PM   #2
kryten2
macrumors 6502a
 
Join Date: Mar 2012
Location: Belgium
Perhaps Little Snitch can help.

Info : http://www.obdev.at/products/littlesnitch/index.html
__________________
Space Corps Directive 34124
kryten2 is offline   0 Reply With Quote
Old Dec 10, 2012, 01:10 AM   #3
nelly22
Thread Starter
macrumors regular
 
Join Date: Sep 2009
Quote:
Originally Posted by kryten2 View Post
Perhaps Little Snitch can help.

Info : http://www.obdev.at/products/littlesnitch/index.html
Thanks but i need to see actual URLs.
nelly22 is offline   0 Reply With Quote
Old Dec 28, 2012, 01:02 AM   #4
nelly22
Thread Starter
macrumors regular
 
Join Date: Sep 2009
What events or notifications or ??? i need to monitor in ASOC to get all URLs visited by Safari? Thanks
nelly22 is offline   0 Reply With Quote
Old Dec 28, 2012, 06:15 AM   #5
MadTester
macrumors regular
 
Join Date: Mar 2012
Quote:
Originally Posted by nelly22 View Post
What events or notifications or ??? i need to monitor in ASOC to get all URLs visited by Safari? Thanks
I might be way off what you are asking but have you tried using the Developer utils in Safari? Web Inspector, Profiler? might this help? also if you expand Little Snitch it does give you a break down as to what is passing through it.

Also if you have your mac firewall on you will be able to view the log.

URL's translate to IP addresses and vice versa. Also if you look in the Utilities > 'Console' you will also see the little snitch network monitor log.

HTH...
__________________
I know what I like and I like what I know...17" Aluminium MacBookPro MountainLion, 8Gb RAM, 320 GB HD, Mac Mini Lion Server 16gb RAM, 1TB HD, Work Unibody MacBookPro 15" 2011 OSX Lion, i7 quad
MadTester is offline   0 Reply With Quote
Old Dec 28, 2012, 11:23 AM   #6
nelly22
Thread Starter
macrumors regular
 
Join Date: Sep 2009
Quote:
Originally Posted by MadTester View Post
I might be way off what you are asking but have you tried using the Developer utils in Safari? Web Inspector, Profiler? might this help? also if you expand Little Snitch it does give you a break down as to what is passing through it.

Also if you have your mac firewall on you will be able to view the log.

URL's translate to IP addresses and vice versa. Also if you look in the Utilities > 'Console' you will also see the little snitch network monitor log.

HTH...
Thanks.

Little Snitch, firewall log etc don't show full URL like this: http:somedomain.com/path/index.html. I think IP cannot be translated to full URL.

I found GURL Watcher but it don't support Mountain Lion.

At this point i need only URLs which Safari visits. I wonder how it does it?
nelly22 is offline   0 Reply With Quote
Old Dec 28, 2012, 12:45 PM   #7
robvas
macrumors 68000
 
Join Date: Mar 2009
Location: USA
You could use TCP dump:

tcpdump -n -A -s1514 src 1.2.3.4 and port 80 | grep "GET\|Host:"

Replace 1.2.3.4 with the IP address of your computer

You'll get stuff like:
ost: cdn.api.twitter.com
:.....ZQGET /uds/css/small-logo.png HTTP/1.1
Host: www.google.com
:.....].GET /uds/css/v2/search_box_icon.png HTTP/1.1
robvas is offline   0 Reply With Quote
Old Dec 28, 2012, 01:41 PM   #8
nelly22
Thread Starter
macrumors regular
 
Join Date: Sep 2009
Quote:
Originally Posted by robvas View Post
You could use TCP dump:

tcpdump -n -A -s1514 src 1.2.3.4 and port 80 | grep "GET\|Host:"

Replace 1.2.3.4 with the IP address of your computer

You'll get stuff like:
ost: cdn.api.twitter.com
:.....ZQGET /uds/css/small-logo.png HTTP/1.1
Host: www.google.com
:.....].GET /uds/css/v2/search_box_icon.png HTTP/1.1
Thanks. That looks exactly what i'm looking for.

Unfortunately i get "tcpdump: no suitable device found" even after replacing 1.2.3.4 with my ip address from network preference pane.

Other thing is can i change it to monitor all ports, not just 80?
nelly22 is offline   0 Reply With Quote
Old Dec 28, 2012, 01:43 PM   #9
MisterKeeks
macrumors 68000
 
MisterKeeks's Avatar
 
Join Date: Nov 2012
Have you tried TCPBlock
MisterKeeks is offline   0 Reply With Quote
Old Dec 28, 2012, 07:53 PM   #10
pitaya
macrumors member
 
Join Date: Jun 2012
Quote:
Originally Posted by nelly22 View Post
Unfortunately i get "tcpdump: no suitable device found"
sudo tcpdump ...

Quote:
Other thing is can i change it to monitor all ports, not just 80?
Yeah, snip off the "and port 80" part of the expression.
pitaya is offline   0 Reply With Quote
Old Dec 28, 2012, 09:48 PM   #11
jared_kipe
macrumors 68030
 
jared_kipe's Avatar
 
Join Date: Dec 2003
Location: Seattle
Send a message via AIM to jared_kipe
TCPDump is only particularly good at getting a snapshot at a specific point in time.

The only dynamic way I now of would be to use a 'hardcore' program like wireshark to intercept all IP traffic and then parse through it looking for whatever you're after. (you'll need Xwindows or something installed, I haven't used it for a while)

Slightly easier might be using 'iftop' from the command line, but you'll need to install it through MacPorts (and have Xcode), and very high throughput will crash it (like a 60+Mbps).

You mentioned Safari. What is wrong with using the built in developer tools->Insturments->Network Requests? (see screenshot)

You can generally see every url (full url on hover). and even see them happen in psudo real time.
Attached Thumbnails
Click image for larger version

Name:	Screen Shot 2012-12-28 at 7.44.52 PM.png
Views:	32
Size:	336.8 KB
ID:	386463  
jared_kipe is offline   0 Reply With Quote
Old Dec 29, 2012, 01:39 AM   #12
nelly22
Thread Starter
macrumors regular
 
Join Date: Sep 2009
Sorry i didn't explain it accurately in the first place. I can't use Safari developer tools, because i need text log file. Log don't need to be nice looking as long it logs.

http://www.quicomm.com/gurl_watcher_help_osx.html

"Have you tried TCPBlock"

No and it looks like it's overkill for my use and i think it don't log full URLs.

"Other thing is can i change it to monitor all ports, not just 80?
Yeah, snip off the "and port 80" part of the expression."

Cool.

"sudo tcpdump ..."

With sudo it works, but date and time stamp is still needed. And if possible, name of application which does this connection. I know nothing about grep.

I would like to also try iftop. What is easiest way to install MacPorts?

Thanks

Quote:
Originally Posted by jared_kipe View Post
TCPDump is only particularly good at getting a snapshot at a specific point in time.

The only dynamic way I now of would be to use a 'hardcore' program like wireshark to intercept all IP traffic and then parse through it looking for whatever you're after. (you'll need Xwindows or something installed, I haven't used it for a while)

Slightly easier might be using 'iftop' from the command line, but you'll need to install it through MacPorts (and have Xcode), and very high throughput will crash it (like a 60+Mbps).

You mentioned Safari. What is wrong with using the built in developer tools->Insturments->Network Requests? (see screenshot)

You can generally see every url (full url on hover). and even see them happen in psudo real time.
nelly22 is offline   0 Reply With Quote
Old Dec 29, 2012, 02:52 AM   #13
web_god61
macrumors regular
 
Join Date: May 2004
lsof |grep TCP | grep ESTAB
web_god61 is offline   0 Reply With Quote
Old Dec 29, 2012, 05:57 AM   #14
MadTester
macrumors regular
 
Join Date: Mar 2012
Posted in error
__________________
I know what I like and I like what I know...17" Aluminium MacBookPro MountainLion, 8Gb RAM, 320 GB HD, Mac Mini Lion Server 16gb RAM, 1TB HD, Work Unibody MacBookPro 15" 2011 OSX Lion, i7 quad
MadTester is offline   0 Reply With Quote
Old Dec 29, 2012, 10:01 AM   #15
jared_kipe
macrumors 68030
 
jared_kipe's Avatar
 
Join Date: Dec 2003
Location: Seattle
Send a message via AIM to jared_kipe
Quote:
Originally Posted by nelly22 View Post
Sorry i didn't explain it accurately in the first place. I can't use Safari developer tools, because i need text log file. Log don't need to be nice looking as long it logs.

http://www.quicomm.com/gurl_watcher_help_osx.html

"Have you tried TCPBlock"

No and it looks like it's overkill for my use and i think it don't log full URLs.

"Other thing is can i change it to monitor all ports, not just 80?
Yeah, snip off the "and port 80" part of the expression."

Cool.

"sudo tcpdump ..."

With sudo it works, but date and time stamp is still needed. And if possible, name of application which does this connection. I know nothing about grep.

I would like to also try iftop. What is easiest way to install MacPorts?

Thanks
iftop won't give you date and time text log output.

Sounds like you need to jump into the deep end with Wireshark.
jared_kipe is offline   0 Reply With Quote
Old Dec 29, 2012, 10:18 AM   #16
pitaya
macrumors member
 
Join Date: Jun 2012
You could write up a script to parse tcpdump output, similar to this:
http://n3t.awardspace.us/content/tcpdump-url-extraction

It would have to be modified for OS X, and you want a timestamp:

Code:
#!/bin/bash
#
 
# reset variables
myhost="";
myurl="";
 
tcpdump -s 0 -w - -l $@ | strings |
while read line;
	do 
 
# filter GET requests
	myurl=`echo $line | grep GET | sed -E "s/GET (.*) HTTP.*/\1/"`;
	if [ "$myurl" == "" ]; then myurl=$myoldurl; fi
 
# filter Host headers
	myhost=`echo $line | grep Host | sed -E "s/Host: (.*)/\1/"`;
	if [ "$myhost" == "" ]; then myhost=$myoldhost; fi
 
# once we have a data pair, put them together and echo
	if [ "$myhost" != "" ] 
		then
		url="http://$myhost$myurl";
		echo -n "$(date): "
		echo $url;
		myhost="";
		myurl="";
	fi
 
	myoldurl=$myurl;
	myoldhost=$myhost;
done
Alternatively, just write out the packets to a file and analyze it later with whatever tool you want (tcpdump's -w flag, -r to read back packets from the file, ethereal/wireshark, etc).
pitaya is offline   0 Reply With Quote
Old Dec 30, 2012, 08:18 AM   #17
nelly22
Thread Starter
macrumors regular
 
Join Date: Sep 2009
Thanks, this looks cool.

I saved your script to plain text file test_fs.sh.

Then i run this in Terminal app:

chmod +x /Users/Nelly/Desktop/test_fs.sh
sudo /Users/Nelly/Desktop/test_fs.sh

I cannot find log file anywhere. I think echo row(s) need something?? It don't have to save data after every url, just now and then.

When i cancel it, i get this:

^C577 packets captured
8060 packets received by filter
7371 packets dropped by kernel

Quote:
Originally Posted by pitaya View Post
You could write up a script to parse tcpdump output, similar to this:
http://n3t.awardspace.us/content/tcpdump-url-extraction

It would have to be modified for OS X, and you want a timestamp:

Code:
#!/bin/bash
#
 
# reset variables
myhost="";
myurl="";
 
tcpdump -s 0 -w - -l $@ | strings |
while read line;
	do 
 
# filter GET requests
	myurl=`echo $line | grep GET | sed -E "s/GET (.*) HTTP.*/\1/"`;
	if [ "$myurl" == "" ]; then myurl=$myoldurl; fi
 
# filter Host headers
	myhost=`echo $line | grep Host | sed -E "s/Host: (.*)/\1/"`;
	if [ "$myhost" == "" ]; then myhost=$myoldhost; fi
 
# once we have a data pair, put them together and echo
	if [ "$myhost" != "" ] 
		then
		url="http://$myhost$myurl";
		echo -n "$(date): "
		echo $url;
		myhost="";
		myurl="";
	fi
 
	myoldurl=$myurl;
	myoldhost=$myhost;
done
Alternatively, just write out the packets to a file and analyze it later with whatever tool you want (tcpdump's -w flag, -r to read back packets from the file, ethereal/wireshark, etc).
nelly22 is offline   0 Reply With Quote
Old Dec 30, 2012, 08:42 AM   #18
pitaya
macrumors member
 
Join Date: Jun 2012
You might want to pipe it through tee, or just append it to a log file:


Code:
sudo /Users/Nelly/Desktop/test_fs.sh | tee -a urls.log
Code:
sudo /Users/Nelly/Desktop/test_fs.sh >> urls.log
pitaya is offline   0 Reply With Quote
Old Dec 30, 2012, 01:13 PM   #19
Q-chan
macrumors member
 
Join Date: Nov 2009
Location: Boston, MA, USA
Snort is your friend

Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.

Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time.

Good luck and Happy New Year,

Manfred
Q-chan is offline   0 Reply With Quote
Old Dec 30, 2012, 05:32 PM   #20
robvas
macrumors 68000
 
Join Date: Mar 2009
Location: USA
Quote:
Originally Posted by Q-chan View Post
Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.

Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time.

Good luck and Happy New Year,

Manfred
Is snort or any other IDS tools available in homebrew?
robvas is offline   0 Reply With Quote
Old Dec 30, 2012, 05:51 PM   #21
Q-chan
macrumors member
 
Join Date: Nov 2009
Location: Boston, MA, USA
Quote:
Originally Posted by robvas View Post
Is snort or any other IDS tools available in homebrew?
I'm not sure (and tend to doubt it). I prefer to use the more traditional approach of "configure --> make --> make install". On Mountain Lion you might need to build autoconf and automake, as they are no longer in Xcode (), but make sure NOT to replace libtool.

You can always try to run the configure script without these tools installed, the script will tell you when a tool is missing.

Manfred
Q-chan is offline   0 Reply With Quote
Old Dec 30, 2012, 06:22 PM   #22
ipsychedelic
macrumors 6502a
 
ipsychedelic's Avatar
 
Join Date: Mar 2012
Location: Cali, Colombia
Wireshark is what I use. Never required anything else, on OS X.
Just make sure, if you decide to use it, to get proficient (30 mins) on usage of filters, so you can filter out garbage you don't wanna "listen to".
ipsychedelic is offline   0 Reply With Quote
Old Dec 30, 2012, 06:33 PM   #23
Q-chan
macrumors member
 
Join Date: Nov 2009
Location: Boston, MA, USA
Quote:
Originally Posted by ipsychedelic View Post
Wireshark is what I use. Never required anything else, on OS X.
Just make sure, if you decide to use it, to get proficient (30 mins) on usage of filters, so you can filter out garbage you don't wanna "listen to".
Good tool, but the OP needs text (text file ?) output, not X11 screens....
Q-chan is offline   0 Reply With Quote
Old Dec 30, 2012, 08:16 PM   #24
ipsychedelic
macrumors 6502a
 
ipsychedelic's Avatar
 
Join Date: Mar 2012
Location: Cali, Colombia
Quote:
Originally Posted by Q-chan View Post
Good tool, but the OP needs text (text file ?) output, not X11 screens....
You're right, I was quick on the response but didn't really bothered to read the whole topic (somebody had suggested Wireshark before anyway).

I think wireshark can be called from the command line, or one can use TShark. But my usage has always been in the GUI.
ipsychedelic is offline   0 Reply With Quote
Old Dec 30, 2012, 08:56 PM   #25
robvas
macrumors 68000
 
Join Date: Mar 2009
Location: USA
Quote:
Originally Posted by Q-chan View Post
I'm not sure (and tend to doubt it). I prefer to use the more traditional approach of "configure --> make --> make install". On Mountain Lion you might need to build autoconf and automake, as they are no longer in Xcode (), but make sure NOT to replace libtool.
I've played enough cat and mouse with libraries and packages over the years. I just checked brew and there's a snort formula availab.e
robvas is offline   0 Reply With Quote

Reply
MacRumors Forums > Apple Systems and Services > Programming > Mac Programming

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Similar Threads
thread Thread Starter Forum Replies Last Post
Embedded URLS and Georiot SaSaSushi Site and Forum Feedback 3 Mar 5, 2014 11:38 AM
Shortened URLs in iOS 7? jason2811 iOS 7 4 Dec 24, 2013 09:57 AM
Email sigs come out as URLs? hakr100 iOS 6 6 Oct 15, 2012 09:32 AM
download locations from different URLs wootalicious Mac OS X 10.7 Lion 4 Sep 5, 2012 11:38 AM
Messages.app URLs? tarryweather OS X 10.8 Mountain Lion 0 Jul 30, 2012 01:28 PM

Forum Jump

All times are GMT -5. The time now is 04:38 AM.

Mac Rumors | Mac | iPhone | iPhone Game Reviews | iPhone Apps

Mobile Version | Fixed | Fluid | Fluid HD
Copyright 2002-2013, MacRumors.com, LLC