Troubleshooting: getting more from kernel panics with keepsyms (address to symbol translation)

Discussion in 'macOS' started by grahamperrin, Jan 1, 2015.

  1. grahamperrin, Jan 1, 2015
    Last edited: Sep 9, 2016

    grahamperrin macrumors 601

    grahamperrin

    Joined:
    Jun 8, 2007
    #1
    Normally the 'Backtrace' part of a report of a kernel panic will present information that's quite unintelligible.

    In this photograph, for example:

    example.png


    For troubleshooting

    Readers are more likely to make sense of a backtrace if you set a boot argument – keepsyms=y – in NVRAM.

    With Terminal, whilst logged in as an administrator, be prepared to enter the administrator's password then enter the following command:

    sudo nvram boot-args="-v keepsyms=y"

    The first of those two boot arguments (-v) causes the operating system to start and shut down in verbose mode.

    For any change to boot arguments to be effective, restart (reboot) the OS.


    With symbols: examples of .panic file content

    Backtraces are presented in reverse chronological order, so read from bottom to top.

    (The examples below exclude the list of loaded kernel extensions – they're irrelevant to this topic.)


    Example A: OS X Mavericks

    This panic occurred during a check of an HFS Plus file system.

    Near the top of the trace there's something obviously related to HFS:

    _hfs_vnop_close +

    Code:
    Anonymous UUID:       CA907937-552D-42D6-3F32-C7D13950F0B4
    
    Thu Oct  2 12:50:29 2014
    panic(cpu 1 caller 0xffffff800938533e): "lck_rw_unlock(): lock held in mode: 2\n"@/SourceCache/xnu/xnu-2422.115.4/osfmk/i386/locks_i386.c:1259
    Backtrace (CPU 1), Frame : Return Address
    0xffffff810d22bb50 : 0xffffff8009022f79 mach_kernel : _panic + 0xc9
    0xffffff810d22bbd0 : 0xffffff800938533e mach_kernel : _hfs_vnop_close + 0x33e
    0xffffff810d22bc30 : 0xffffff80091fe02e mach_kernel : _VNOP_CLOSE + 0xce
    0xffffff810d22bca0 : 0xffffff80091f539b mach_kernel : _vn_close + 0x7b
    0xffffff810d22bce0 : 0xffffff80091f45bf mach_kernel : _utf8_normalizestr + 0xbbf
    0xffffff810d22bd30 : 0xffffff80093c1dec mach_kernel : _closef_locked + 0x13c
    0xffffff810d22bda0 : 0xffffff80093c3adb mach_kernel : _fdfree + 0x10b
    0xffffff810d22bde0 : 0xffffff80093cf232 mach_kernel : _proc_exit + 0x1d2
    0xffffff810d22be60 : 0xffffff80090456b9 mach_kernel : _thread_terminate_self + 0x119
    0xffffff810d22be90 : 0xffffff8009048789 mach_kernel : _special_handler + 0xa9
    0xffffff810d22bec0 : 0xffffff80090485ee mach_kernel : _act_execute_returnhandlers + 0x7e
    0xffffff810d22bef0 : 0xffffff8009020b63 mach_kernel : _ast_taken + 0xe3
    0xffffff810d22bf20 : 0xffffff80090dca73 mach_kernel : _i386_astintr + 0x23
    0xffffff810d22bf40 : 0xffffff80090f3942 mach_kernel : _return_from_trap + 0xb2
    
    BSD process name corresponding to current thread: fsck_hfs
    Boot args: -v keepsyms=y
    
    Mac OS version:
    13F34
    
    Kernel version:
    Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64
    Kernel UUID: 9477416E-7BCA-3679-AF97-E1EAAD3DD5A0
    Kernel slide:     0x0000000008e00000
    Kernel text base: 0xffffff8009000000
    System model name: MacBookPro5,2 (Mac-F2268EC8)
    
    System uptime in nanoseconds: 56004494853809
    last loaded kext at 55900712405425: com.apple.driver.AppleFireWireStorage    3.4.1 (addr 0xffffff7f8b90d000, size 16384)
    last unloaded kext at 41329547225938: com.apple.driver.AppleUSBCDC    4.2.1b5 (addr 0xffffff7f8b82b000, size 16384)
    loaded kexts:
    
    …
    
    Model: MacBookPro5,2, BootROM MBP52.008E.B05, 2 processors, Intel Core 2 Duo, 2.66 GHz, 8 GB, SMC 1.42f4
    Graphics: NVIDIA GeForce 9400M, NVIDIA GeForce 9400M, PCI, 256 MB
    Graphics: NVIDIA GeForce 9600M GT, NVIDIA GeForce 9600M GT, PCIe, 512 MB
    Memory Module: BANK 0/DIMM0, 4 GB, DDR3, 1067 MHz, 0x80AD, 0x484D5434353153364D4D5238432D47372020
    Memory Module: BANK 1/DIMM0, 4 GB, DDR3, 1067 MHz, 0x80AD, 0x484D5434353153364D4D5238432D47372020
    AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x8D), Broadcom BCM43xx 1.0 (5.106.98.100.22)
    Bluetooth: Version 4.2.7f3 14616, 3 services, 23 devices, 1 incoming serial ports
    Network Service: Ethernet, Ethernet, en0
    Network Service: Wi-Fi, AirPort, en1
    Serial ATA Device: ST750LX003-1AC154, 750.16 GB
    Serial ATA Device: HL-DT-ST DVDRW  GS21N
    USB Device: Built-in iSight
    USB Device: STORE N GO
    USB Device: Apple Internal Keyboard / Trackpad
    USB Device: IR Receiver
    USB Device: Hub in Apple Pro Keyboard
    USB Device: Apple Pro Keyboard
    USB Device: BRCM2046 Hub
    USB Device: Bluetooth USB Host Controller
    FireWire Device: fw_target_disk_mode, AAPL, Up to 800 Mb/sec
    Thunderbolt Bus: 

    Example B: OS X Yosemite

    In this case, the cause of the panic was initially unknown.

    Within the backtrace there's a distinctive string:

    _NodesAreContiguous

    That string was recognised by someone in Ask Different; the panic was associated with an HFS Plus file system where journalling was unexpectedly disabled.

    Code:
    Anonymous UUID:       936245CB-C37F-6300-8568-D67DB990D759
    
    Wed Nov  5 14:16:23 2014
    
    *** Panic Report ***
    panic(cpu 3 caller 0xffffff801c61e80a): Kernel trap at 0xffffff801c30e11b, type 14=page fault, registers:
    CR0: 0x000000008001003b, CR2: 0xffffff82068b4000, CR3: 0x000000001f225000, CR4: 0x00000000001626e0
    RAX: 0x0000000000000000, RBX: 0xffffff82068b3f48, RCX: 0x0000000076fdd000, RDX: 0x0000000000000000
    RSP: 0xffffff82068b3e98, RBP: 0xffffff82068b3ed0, RSI: 0x00000003b7ee8080, RDI: 0xffffff82068b4000
    R8:  0x0000000000000000, R9:  0xffffff82069c5000, R10: 0xffffff80202c8f00, R11: 0x0000000d273b7a85
    R12: 0x000000000000000c, R13: 0x0000000000000000, R14: 0xffffff803b7ee808, R15: 0xffffff82068b3ea0
    RFL: 0x0000000000010202, RIP: 0xffffff801c30e11b, CS:  0x0000000000000008, SS:  0x0000000000000010
    Fault CR2: 0xffffff82068b4000, Error code: 0x0000000000000002, Fault CPU: 0x3
    
    Backtrace (CPU 3), Frame : Return Address
    0xffffff82068b3b40 : 0xffffff801c53a811 mach_kernel : _panic + 0xd1
    0xffffff82068b3bc0 : 0xffffff801c61e80a mach_kernel : _kernel_trap + 0x84a
    0xffffff82068b3d80 : 0xffffff801c63a443 mach_kernel : _return_from_trap + 0xe3
    0xffffff82068b3da0 : 0xffffff801c30e11b
    0xffffff82068b3ed0 : 0xffffff801c97d684 mach_kernel : _NodesAreContiguous + 0x1794
    0xffffff82068b3f80 : 0x0
    
    BSD process name corresponding to current thread: kernel_task
    Boot args: -v keepsyms=y
    
    Mac OS version:
    14A389
    
    Kernel version:
    Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64
    Kernel UUID: 89E10306-BC78-3A3B-955C-7C4922577E61
    Kernel slide:     0x000000001c200000
    Kernel text base: 0xffffff801c400000
    __HIB  text base: 0xffffff801c300000
    System model name: MacBookPro11,1 (Mac-189A3D4F975D5FFC)
    
    System uptime in nanoseconds: 2061441990
    last loaded kext at 1497836320: com.apple.driver.CoreStorageFsck    471 (addr 0xffffff7f9e041000, size 106496)
    loaded kexts:
    
    …
    

    Related

    nvram(8) Mac OS X Manual Page in the Apple Developer Retired Documents Library

    How to Reset NVRAM on your Mac – Apple Support

    xnu-1228 / Boot Arguments - Tutorials (The Genius Bar) - InsanelyMac Forum (2008-05-25, and captured in the Wayback Machine)

    http://fxr.watson.org/fxr/source/osfmk/i386/AT386/model_dep.c?v=xnu-1228#L818

    Postscript

    The photograph that was originally with this post was lost. Now in its place: an edition of the photograph from example B.
     
  2. grahamperrin thread starter macrumors 601

    grahamperrin

    Joined:
    Jun 8, 2007
    #2
    With more modern versions of Mac OS X, where system Integrity protection (SIP) is enabled

    Whilst OS X or macOS is running normally, SIP prevents the required setting.

    To make the setting:
    1. start Recovery OS
    2. use the Utilities menu to launch Terminal
    3. enter the command below, then restart the operating system.
    nvram boot-args="-v keepsyms=y"


    Questions and answers

    Someone asked:

    Yes, that's verbose mode, the -v part of the command.

    I never tried that but it seems logical. n insteady of y

    Also you can reset NVRAM, which will lose both boot arguments.
     

Share This Page