Reverse Engineering pkgdmg Files

[toru173]

Edit: update in the next post. Utility and writeup available

Hi All,

I wanted to drop some notes here about a recent project. I've been trying to reverse engineer the pkgdmg format as I'm trying to build a custom installer for my older macs. This may be of interest to those that want to run macOS on unsupported macs; I know @ASentientBot mentioned the format a while ago in this comment.

A pkgdmg is a combination file that acts as a dmg with the extension .dmg, and as a macOS installation file (a package) with the extension .pkg. The only thing you need to do to change the behaviour of the file is to change the extension. This works because packages are effectively xar archives (good primer here) which don't care about any data that isn't in the file's XML-based Table of Contents. The pkgdmg file format uses this as an advantage as it simply concatenates dmg-relevant information (a 'koly block', good primer here) on to the end of the xar file, which is ignored when the OS treats the pkgdmg as a pkg file.

What I've found so far:

• A pkgdmg is essentially a dmg enclosed in an (optionally) signed xar file, with supporting files expected per a distribution-style package (good primer here)
• The files in a pkgdmg are not compressed. Though parts may be - such as the dmg itself - the significant files are uncompressed as the system needs to know exact byte offset within the pkgdmg to make things work, and they need to be able to be read byte-for-byte from the xar archive
• The inner dmg at /Archive/of/OS/installer/OSInstall.pkg/InstallESD.dmg appears to be a normal, zlib-compressed dmg - one that we can create with hdiutil

So far I've had success in manually extracting then re-compressing a pkgdmg using the following process:

mkdir InstallESD
cd InstallESD
sudo xar -x -P -f ../InstallESD.dmg
< Edits to InstallOS.pkg/InstallESD.dmg go here >
sudo xar -c --compression=none -f ../New\ InstallESD.pkg *
dd if=./InstallOS.pkg/InstallESD.dmg skip=[IF_EOF-512] bs=1 of=../InstallESD\ koly\ from\ dmg.bin
cd ..
< Edit Install\ ESD\ Koly\ from\ dmg.bin to correct where the start of the dmg is in the final xar archive >
sudo dd if=new\ InstallESD\ koly.bin of=New\ InstallESD.pkg bs=1 seek=[OF_EOF]
mv InstallESD.pkg InstallESD.dmg

This is really abridged and I hope to build a more in-depth explanation in the future with screenshots and maybe a user friendly tool to automate the binary editing, but that will have to wait. At the moment the macOS installer accepts my modified pkgdmg and doesn't give the dreaded pkgdmg is missing a footer/pkgdmg validation has failed errors, but it fails a signature check. This is expected as I stripped out signatures completely!

Let me know if this is interesting, and I'll keep posting my updates. When complete I'll do up a page on my GitHub (https://github.com/toru173) with my research and any automation tools I create.

 

[toru173]

I've written a utility to automate the process. More information and some of my notes are available at my GitHub page (https://github.com/toru173/pkg2pkgdmg).

Using the utility, editing a pkgdmg (eg InstallESD.dmg) is as simple as follows:

cp /path/to/InstallESD.dmg .
mkdir InstallESD
cd InstallESD
sudo xar -x -P -f ../InstallESD.dmg
< Edits to InstallOS.pkg/InstallESD.dmg go here >
sudo xar -c --compression=none -f ../New\ InstallESD.pkg *
cd ..
sudo ./pkg2pkgdmg New\ InstallESD.pkg New\ InstallESD.dmg