Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
macOS Monitor URLs
- Thread starter nelly22
- Start date
- Sort by reaction score
I might be way off what you are asking but have you tried using the Developer utils in Safari? Web Inspector, Profiler? might this help? also if you expand Little Snitch it does give you a break down as to what is passing through it.
Also if you have your mac firewall on you will be able to view the log.
URL's translate to IP addresses and vice versa. Also if you look in the Utilities > 'Console' you will also see the little snitch network monitor log.
HTH...
Thanks.
Little Snitch, firewall log etc don't show full URL like this: http:somedomain.com/path/index.html. I think IP cannot be translated to full URL.
I found GURL Watcher but it don't support Mountain Lion.
At this point i need only URLs which Safari visits. I wonder how it does it?
You could use TCP dump:
tcpdump -n -A -s1514 src 1.2.3.4 and port 80 | grep "GET\|Host:"
Replace 1.2.3.4 with the IP address of your computer
You'll get stuff like:
ost: cdn.api.twitter.com
:.....ZQGET /uds/css/small-logo.png HTTP/1.1
Host: http://www.google.com
:.....].GET /uds/css/v2/search_box_icon.png HTTP/1.1
tcpdump -n -A -s1514 src 1.2.3.4 and port 80 | grep "GET\|Host:"
Replace 1.2.3.4 with the IP address of your computer
You'll get stuff like:
ost: cdn.api.twitter.com
:.....ZQGET /uds/css/small-logo.png HTTP/1.1
Host: http://www.google.com
:.....].GET /uds/css/v2/search_box_icon.png HTTP/1.1
Thanks. That looks exactly what i'm looking for.
Unfortunately i get "tcpdump: no suitable device found" even after replacing 1.2.3.4 with my ip address from network preference pane.
Other thing is can i change it to monitor all ports, not just 80?
TCPDump is only particularly good at getting a snapshot at a specific point in time.
The only dynamic way I now of would be to use a 'hardcore' program like wireshark to intercept all IP traffic and then parse through it looking for whatever you're after. (you'll need Xwindows or something installed, I haven't used it for a while)
Slightly easier might be using 'iftop' from the command line, but you'll need to install it through MacPorts (and have Xcode), and very high throughput will crash it (like a 60+Mbps).
You mentioned Safari. What is wrong with using the built in developer tools->Insturments->Network Requests? (see screenshot)
You can generally see every url (full url on hover). and even see them happen in psudo real time.
The only dynamic way I now of would be to use a 'hardcore' program like wireshark to intercept all IP traffic and then parse through it looking for whatever you're after. (you'll need Xwindows or something installed, I haven't used it for a while)
Slightly easier might be using 'iftop' from the command line, but you'll need to install it through MacPorts (and have Xcode), and very high throughput will crash it (like a 60+Mbps).
You mentioned Safari. What is wrong with using the built in developer tools->Insturments->Network Requests? (see screenshot)
You can generally see every url (full url on hover). and even see them happen in psudo real time.
Attachments
Sorry i didn't explain it accurately in the first place. I can't use Safari developer tools, because i need text log file. Log don't need to be nice looking as long it logs.
http://www.quicomm.com/gurl_watcher_help_osx.html
"Have you tried TCPBlock"
No and it looks like it's overkill for my use and i think it don't log full URLs.
"Other thing is can i change it to monitor all ports, not just 80?
Yeah, snip off the "and port 80" part of the expression."
Cool.
"sudo tcpdump ..."
With sudo it works, but date and time stamp is still needed. And if possible, name of application which does this connection. I know nothing about grep.
I would like to also try iftop. What is easiest way to install MacPorts?
Thanks
http://www.quicomm.com/gurl_watcher_help_osx.html
"Have you tried TCPBlock"
No and it looks like it's overkill for my use and i think it don't log full URLs.
"Other thing is can i change it to monitor all ports, not just 80?
Yeah, snip off the "and port 80" part of the expression."
Cool.
"sudo tcpdump ..."
With sudo it works, but date and time stamp is still needed. And if possible, name of application which does this connection. I know nothing about grep.
I would like to also try iftop. What is easiest way to install MacPorts?
Thanks
iftop won't give you date and time text log output.
Sounds like you need to jump into the deep end with Wireshark.
You could write up a script to parse tcpdump output, similar to this:
http://n3t.awardspace.us/content/tcpdump-url-extraction
It would have to be modified for OS X, and you want a timestamp:
Alternatively, just write out the packets to a file and analyze it later with whatever tool you want (tcpdump's -w flag, -r to read back packets from the file, ethereal/wireshark, etc).
http://n3t.awardspace.us/content/tcpdump-url-extraction
It would have to be modified for OS X, and you want a timestamp:
Code:
#!/bin/bash
#
# reset variables
myhost="";
myurl="";
tcpdump -s 0 -w - -l $@ | strings |
while read line;
do
# filter GET requests
myurl=`echo $line | grep GET | sed -E "s/GET (.*) HTTP.*/\1/"`;
if [ "$myurl" == "" ]; then myurl=$myoldurl; fi
# filter Host headers
myhost=`echo $line | grep Host | sed -E "s/Host: (.*)/\1/"`;
if [ "$myhost" == "" ]; then myhost=$myoldhost; fi
# once we have a data pair, put them together and echo
if [ "$myhost" != "" ]
then
url="http://$myhost$myurl";
echo -n "$(date): "
echo $url;
myhost="";
myurl="";
fi
myoldurl=$myurl;
myoldhost=$myhost;
doneAlternatively, just write out the packets to a file and analyze it later with whatever tool you want (tcpdump's -w flag, -r to read back packets from the file, ethereal/wireshark, etc).
Thanks, this looks cool.
I saved your script to plain text file test_fs.sh.
Then i run this in Terminal app:
chmod +x /Users/Nelly/Desktop/test_fs.sh
sudo /Users/Nelly/Desktop/test_fs.sh
I cannot find log file anywhere. I think echo row(s) need something?? It don't have to save data after every url, just now and then.
When i cancel it, i get this:
^C577 packets captured
8060 packets received by filter
7371 packets dropped by kernel
I saved your script to plain text file test_fs.sh.
Then i run this in Terminal app:
chmod +x /Users/Nelly/Desktop/test_fs.sh
sudo /Users/Nelly/Desktop/test_fs.sh
I cannot find log file anywhere. I think echo row(s) need something?? It don't have to save data after every url, just now and then.
When i cancel it, i get this:
^C577 packets captured
8060 packets received by filter
7371 packets dropped by kernel
Snort is your friend
Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.
Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time.
Good luck and Happy New Year,
Manfred
Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.
Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time.
Good luck and Happy New Year,
Manfred
Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.
Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time.
Good luck and Happy New Year,
Manfred
Is snort or any other IDS tools available in homebrew?
I'm not sure (and tend to doubt it). I prefer to use the more traditional approach of "configure --> make --> make install". On Mountain Lion you might need to build autoconf and automake, as they are no longer in Xcode
mad, but make sure NOT to replace libtool.You can always try to run the configure script without these tools installed, the script will tell you when a tool is missing.
Manfred
You're right, I was quick on the response but didn't really bothered to read the whole topic (somebody had suggested Wireshark before anyway).
I think wireshark can be called from the command line, or one can use TShark. But my usage has always been in the GUI.
I'm not sure (and tend to doubt it). I prefer to use the more traditional approach of "configure --> make --> make install". On Mountain Lion you might need to build autoconf and automake, as they are no longer in Xcode
I've played enough cat and mouse with libraries and packages over the years. I just checked brew and there's a snort formula availab.e