Logging internet bandwidth with Airport Extreme

Discussion in 'Mac OS X Server, Xserve, and Networking' started by getrealbro, Dec 22, 2015.

  1. getrealbro, Dec 22, 2015
    Last edited: Dec 22, 2015

    getrealbro macrumors regular

    Joined:
    Sep 25, 2015
    #1
    I’m trying to log the internet bandwidth I use each month. My Airport Extreme is an older model that supports SNMP. These two terminal commands extract the correct counters for downloads and uploads (in bytes).

    snmpget -v 2c -c public Hostname ifInOctets.2
    snmpget -v 2c -c public Hostname ifOutOctets.2

    The results look like this

    IF-MIB::ifInOctets.2 = Counter32: 3994037720
    IF-MIB::ifOutOctets.2 = Counter32: 2161993538

    I would appreciate help automating a log of the extracted data with a shell script that would periodically run both commands and append the results to a text file with the current time.

    It would be nice for the resulting text file to be in the following format: dd/mm/yy DownloadCounter UploadCounter. This would make the data easier to load into a spreadsheet for analysis/graphing. But I don't mind cleaning up the log manually if necessary.

    FWIW I already own NetUse Traffic Monitor which tells me the downloads and uploads in real time and will report a cumulative total. But it does not maintain a record of these data that can be analyzed/graphed.

    Thanks in advance.

    —GetRealBro
     
  2. getrealbro thread starter macrumors regular

    Joined:
    Sep 25, 2015
    #2
    Here's my first stab at a Terminal command that produces a log of the current value of my Airport Extreme's download counter every minute.

    while true; do snmpget -v 2c -c public SH-Extreme.local ifInOctets.2 | while read line ; do echo -e "$(date +%H:%M:%S)\t $line" ; done ; sleep 60; done >> APlog2.txt

    The resulting log requires a little manual cleanup in Text Wangler before importing to Excel. But it works :)

    22:45:22 IF-MIB::ifInOctets.2 = Counter32: 3026571806
    22:46:23 IF-MIB::ifInOctets.2 = Counter32: 3026986142
    22:47:23 IF-MIB::ifInOctets.2 = Counter32: 3027811370
    22:48:23 IF-MIB::ifInOctets.2 = Counter32: 3028436918
    22:49:23 IF-MIB::ifInOctets.2 = Counter32: 3028863908
    22:50:23 IF-MIB::ifInOctets.2 = Counter32: 3029363584

    I could still use a little scripting help to clean up the output and include the upload counter on the same line.

    ---GetRealBro
     
  3. getrealbro thread starter macrumors regular

    Joined:
    Sep 25, 2015
    #3
    Here’s v2 of the terminal command which retrieves the download and upload counters (in bytes) from my AirPort Extreme every 10 minutes. Note: this command creates an infinite loop that you stop with cntrl-C.

    while true; do echo -e $(snmpget -v 2c -c public SH-Extreme.local ifInOctets.2)'\t' $(snmpget -v 2c -c public SH-Extreme.local ifOutOctets.2) | while read line ; do echo -e "$(date "+%Y-%m-%d \t%H:%M:%S")\t $line" | sed 's/IF-MIB::ifInOctets.2 = Counter32://' | sed 's/IF-MIB::ifOutOctets.2 = Counter32://'; done; sleep 600; done >> AirPortLog.txt
    The resulting AirPortLog.txt looks like this: Date, Time, Download Counter, Upload Counter (in bytes).

    2015-12-23 21:40:02 2351170846 1778234690
    2015-12-23 21:50:02 2357373810 1846660420
    2015-12-23 22:00:02 2365024870 1916433456
    2015-12-23 22:10:02 2371788150 1982068200
    2015-12-23 22:20:02 2379356720 2040425804​

    It’s fairly easy to import these tab separated values into the spreadsheet of your choice and graph the download/upload volume during any given period e.g. sample interval, day, week, month, etc..

    Note: These raw octet data must be processed a little to compensate for how the AirPort handles the raw 32 bit counters. And this terminal command only works on older AirPorts, because Apple dropped the SNMP capability from its current crop of tower shaped AirPorts.

    If you are interested in more details, post a reply.

    —GetRealBro
     
  4. SailorMoon5 macrumors newbie

    Joined:
    Dec 23, 2015
    Location:
    Florida
    #4
    I'm not sure who to ask, or where to go...Is this normal keychain storage logs?

    Apple Persistent State Encryption
    Call History User Data Key
    com.apple.ids: DeviceHBI
    ids: identity-rsa-private-key
    iTunes iAd password
    Safari Extensions List
    Safari WebCrypto Master Key

    I didn't create any of these.
    normal?
     
  5. getrealbro, Dec 24, 2015
    Last edited: Dec 24, 2015

    getrealbro thread starter macrumors regular

    Joined:
    Sep 25, 2015
    #5
    Here is a graph of the 1st 24 hrs of AirPort monitoring beginning yesterday at 8AM. The red bars are the average download throughput in KB/sec during each sample period (10 minutes). The blue line is the cumulative downloads in MB since the log began.

    AirPortGraph.png

    I’ve already learned quite a bit about our internet bandwidth/usage.
    * The 1st two red blips (approx 64 KB/s) are when I was updating a few iOS apps via iTunes.
    * The larger group is when we were streaming Netflix video in the evening.
    * The last pair were while we were on the web reading news/comics etc. the next morning.

    This last pair (approx 192KB/Sec) is much too big for just a little morning web browsing. You can see that traffic on either side of the two larger red bars. After a little investigation I realized that these two bars also include when Apple automatically downloaded the iOS 9.2 update to our iPad. This confirms my suspicion that Apple downloads these automatic iOS updates at their convenience, not ours :(

    —GetRealBro
     
  6. adam9c1 macrumors 65816

    adam9c1

    Joined:
    May 2, 2012
    Location:
    Chicagoland
    #6
    What tool did you use to graph?

    FYI I used two snmp apps to monitor my AE (from the app store). They were both running at the same time and they were giving me different results during my test.

    I needed to test how much data IN and OUT when I was updating an iPad.

    I ended up pulling this info instead from my wireless access point (I have wifi turned off on the Airport)
     
  7. getrealbro thread starter macrumors regular

    Joined:
    Sep 25, 2015
    #7
    adam9c1:

    The graphing is done in Excel. This can be semi automated if you build the graphs using named ranges that cover a few more rows than the current data. I just insert a bunch of new rows within the graph data range, then copy and paste the new data from the log. Then I drag down a few columns of formulas that convert the raw data to cumulative MB and KB/sec in the graph.

    I only posted an example of the download graph. The upload graph is often more interesting because it captures when our iOS devices are doing automatic iCloud backups. I previously assumed that iOS devices would do their iCloud backups at night. But that isn’t the case. The iOS devices will kick off their backups at any time. Even when our slow internet connection is struggling to stream Netflix without buffering :( No wonder iCloud backups frequently fail.

    The Airport Extreme I’m monitoring is our wireless access point. So the raw data is our total traffic. I checked the values by downloading a large file and comparing to the log. BTW the Airport Extreme reports twice the actual traffic and the 32 bit counters wrap back to zero every 4GB. Some network monitoring apps may not correct for these quirks.

    ---GetRealBro
     
  8. getrealbro, Jan 2, 2016
    Last edited: Jan 3, 2016

    getrealbro thread starter macrumors regular

    Joined:
    Sep 25, 2015
    #8
    Since my last post I converted the bash command into a bash script. This let me set up a cron job that collects the incoming and outgoing data on the AirPort Extreme. The cron job runs every 10 minutes (*/10 * * * * ~/AirPortLog.sh).

    I also monitor the incoming and outgoing data on our Macs using the same SNMP technique. Since I run the scripts using the same cron timing I get logs that are synchronized. On our home network, with only a few Macs and iDevices, I can usually deduce which device was up/down loading data via the AirPort Extreme.

    These logs are how I recognized that every night my iPhone was desperately trying to backup to the iCloud but not being successful.

    RecentDLUL.png

    This chart of the recent downloads (red bars, left scale) and uploads (blue bars, right scale) has a sample interval of 10 minutes. Those clusters of blue bars at 35MB/10min, translate into over GB of data being uploaded by my iPhone each night as it tried, unsuccessfully, to do an incremental iCloud backup of roughly 45MB. Ironically the full backup for this iPhone is only around 500MB.

    —GetRealBro
     

Share This Page