Monitor URLs

Discussion in 'Mac Programming' started by nelly22, Dec 9, 2012.

  1. macrumors regular

    Joined:
    Sep 29, 2009
    #1
    Now and then i notice that some Mac apps connect to net.

    I try to find CLI app (or regular app), which shows what URLs these apps tries to connect. Thanks
     
  2. macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #2
  3. thread starter macrumors regular

    Joined:
    Sep 29, 2009
    #3
  4. thread starter macrumors regular

    Joined:
    Sep 29, 2009
    #4
    What events or notifications or ??? i need to monitor in ASOC to get all URLs visited by Safari? Thanks
     
  5. macrumors regular

    MadTester

    Joined:
    Mar 24, 2012
    #5
    I might be way off what you are asking but have you tried using the Developer utils in Safari? Web Inspector, Profiler? might this help? also if you expand Little Snitch it does give you a break down as to what is passing through it.

    Also if you have your mac firewall on you will be able to view the log.

    URL's translate to IP addresses and vice versa. Also if you look in the Utilities > 'Console' you will also see the little snitch network monitor log.

    HTH...
     
  6. thread starter macrumors regular

    Joined:
    Sep 29, 2009
    #6
    Thanks.

    Little Snitch, firewall log etc don't show full URL like this: http:somedomain.com/path/index.html. I think IP cannot be translated to full URL.

    I found GURL Watcher but it don't support Mountain Lion.

    At this point i need only URLs which Safari visits. I wonder how it does it?
     
  7. macrumors 68000

    Joined:
    Mar 29, 2009
    Location:
    USA
    #7
    You could use TCP dump:

    tcpdump -n -A -s1514 src 1.2.3.4 and port 80 | grep "GET\|Host:"

    Replace 1.2.3.4 with the IP address of your computer

    You'll get stuff like:
    ost: cdn.api.twitter.com
    :.....ZQGET /uds/css/small-logo.png HTTP/1.1
    Host: www.google.com
    :.....].GET /uds/css/v2/search_box_icon.png HTTP/1.1
     
  8. thread starter macrumors regular

    Joined:
    Sep 29, 2009
    #8
    Thanks. That looks exactly what i'm looking for.

    Unfortunately i get "tcpdump: no suitable device found" even after replacing 1.2.3.4 with my ip address from network preference pane.

    Other thing is can i change it to monitor all ports, not just 80?
     
  9. macrumors 68000

    MisterKeeks

    Joined:
    Nov 15, 2012
    #9
  10. macrumors member

    Joined:
    Jun 17, 2012
    #10
    sudo tcpdump ...

    Yeah, snip off the "and port 80" part of the expression.
     
  11. macrumors 68030

    jared_kipe

    Joined:
    Dec 8, 2003
    Location:
    Seattle
    #11
    TCPDump is only particularly good at getting a snapshot at a specific point in time.

    The only dynamic way I now of would be to use a 'hardcore' program like wireshark to intercept all IP traffic and then parse through it looking for whatever you're after. (you'll need Xwindows or something installed, I haven't used it for a while)

    Slightly easier might be using 'iftop' from the command line, but you'll need to install it through MacPorts (and have Xcode), and very high throughput will crash it (like a 60+Mbps).

    You mentioned Safari. What is wrong with using the built in developer tools->Insturments->Network Requests? (see screenshot)

    You can generally see every url (full url on hover). and even see them happen in psudo real time.
     

    Attached Files:

  12. thread starter macrumors regular

    Joined:
    Sep 29, 2009
    #12
    Sorry i didn't explain it accurately in the first place. I can't use Safari developer tools, because i need text log file. Log don't need to be nice looking as long it logs.

    http://www.quicomm.com/gurl_watcher_help_osx.html

    "Have you tried TCPBlock"

    No and it looks like it's overkill for my use and i think it don't log full URLs.

    "Other thing is can i change it to monitor all ports, not just 80?
    Yeah, snip off the "and port 80" part of the expression."

    Cool.

    "sudo tcpdump ..."

    With sudo it works, but date and time stamp is still needed. And if possible, name of application which does this connection. I know nothing about grep.

    I would like to also try iftop. What is easiest way to install MacPorts?

    Thanks

     
  13. macrumors regular

    web_god61

    Joined:
    May 14, 2004
    #13
    lsof |grep TCP | grep ESTAB
     
  14. macrumors regular

    MadTester

    Joined:
    Mar 24, 2012
    #14
    Posted in error
     
  15. macrumors 68030

    jared_kipe

    Joined:
    Dec 8, 2003
    Location:
    Seattle
    #15
    iftop won't give you date and time text log output.

    Sounds like you need to jump into the deep end with Wireshark.
     
  16. macrumors member

    Joined:
    Jun 17, 2012
    #16
    You could write up a script to parse tcpdump output, similar to this:
    http://n3t.awardspace.us/content/tcpdump-url-extraction

    It would have to be modified for OS X, and you want a timestamp:

    Code:
    #!/bin/bash
    #
     
    # reset variables
    myhost="";
    myurl="";
     
    tcpdump -s 0 -w - -l $@ | strings |
    while read line;
    	do 
     
    # filter GET requests
    	myurl=`echo $line | grep GET | sed -E "s/GET (.*) HTTP.*/\1/"`;
    	if [ "$myurl" == "" ]; then myurl=$myoldurl; fi
     
    # filter Host headers
    	myhost=`echo $line | grep Host | sed -E "s/Host: (.*)/\1/"`;
    	if [ "$myhost" == "" ]; then myhost=$myoldhost; fi
     
    # once we have a data pair, put them together and echo
    	if [ "$myhost" != "" ] 
    		then
    		url="http://$myhost$myurl";
    		echo -n "$(date): "
    		echo $url;
    		myhost="";
    		myurl="";
    	fi
     
    	myoldurl=$myurl;
    	myoldhost=$myhost;
    done
    Alternatively, just write out the packets to a file and analyze it later with whatever tool you want (tcpdump's -w flag, -r to read back packets from the file, ethereal/wireshark, etc).
     
  17. thread starter macrumors regular

    Joined:
    Sep 29, 2009
    #17
    Thanks, this looks cool.

    I saved your script to plain text file test_fs.sh.

    Then i run this in Terminal app:

    chmod +x /Users/Nelly/Desktop/test_fs.sh
    sudo /Users/Nelly/Desktop/test_fs.sh

    I cannot find log file anywhere. I think echo row(s) need something?? It don't have to save data after every url, just now and then.

    When i cancel it, i get this:

    ^C577 packets captured
    8060 packets received by filter
    7371 packets dropped by kernel

     
  18. macrumors member

    Joined:
    Jun 17, 2012
    #18
    You might want to pipe it through tee, or just append it to a log file:


    Code:
    sudo /Users/Nelly/Desktop/test_fs.sh | tee -a urls.log
    Code:
    sudo /Users/Nelly/Desktop/test_fs.sh >> urls.log
     
  19. macrumors member

    Joined:
    Nov 2, 2009
    Location:
    Boston, MA, USA
    #19
    Snort is your friend

    Snort, the de-facto standard network intrusion tool will serve your needs. You can get it from http://www.snort.org but you have to build it from source. The other caveat is the learning curve. As with most high-power tools, it takes some good study time to make it do what you want.

    Building and operation on Mountain Lion is without problems. Just make sure to build all the support libraries. And if you are snowed-in like me, then the included 249 pages documentation might help you pass the time. ;)

    Good luck and Happy New Year,

    Manfred
     
  20. macrumors 68000

    Joined:
    Mar 29, 2009
    Location:
    USA
    #20
    Is snort or any other IDS tools available in homebrew?
     
  21. macrumors member

    Joined:
    Nov 2, 2009
    Location:
    Boston, MA, USA
    #21
    I'm not sure (and tend to doubt it). I prefer to use the more traditional approach of "configure --> make --> make install". On Mountain Lion you might need to build autoconf and automake, as they are no longer in Xcode :)mad:), but make sure NOT to replace libtool.

    You can always try to run the configure script without these tools installed, the script will tell you when a tool is missing.

    Manfred
     
  22. macrumors 6502a

    ipsychedelic

    Joined:
    Mar 8, 2012
    Location:
    Cali, Colombia
    #22
    Wireshark is what I use. Never required anything else, on OS X.
    Just make sure, if you decide to use it, to get proficient (30 mins) on usage of filters, so you can filter out garbage you don't wanna "listen to".
     
  23. macrumors member

    Joined:
    Nov 2, 2009
    Location:
    Boston, MA, USA
    #23
    Good tool, but the OP needs text (text file ?) output, not X11 screens....
     
  24. macrumors 6502a

    ipsychedelic

    Joined:
    Mar 8, 2012
    Location:
    Cali, Colombia
    #24
    You're right, I was quick on the response but didn't really bothered to read the whole topic (somebody had suggested Wireshark before anyway).

    I think wireshark can be called from the command line, or one can use TShark. But my usage has always been in the GUI.
     
  25. macrumors 68000

    Joined:
    Mar 29, 2009
    Location:
    USA
    #25
    I've played enough cat and mouse with libraries and packages over the years. I just checked brew and there's a snort formula availab.e
     

Share This Page