Can we download Mavericks directly from Apple's servers?

[Wowfunhappy]

^ I still don't see a bswap being done between CID & cid-in-K anywhere. Does this work reliably?

No, it does not work reliably. However, if I do a bswap, it never works! For the life of me, I don't understand what is going on with that. I do seem to end up with the same values of K between the shell and objective-C code, as best as I can tell.

 

[Wowfunhappy]

Also remind me again what the difference is between InstallESD & BaseSystem? I thought InstallESD had the "install osx.app" which had BaseSystem embedded inside of it (last time I analyzed this I remember that it did something clever where it uses an obscure kernel flag to effectively mount and chroot the core system libs). But it seems like you're doing the reverse, putting InstallESD inside base system?

BaseSystem has everything—including the installation app—except for the directory /System/Installation/Packages. Where this folder should be, BaseSystem just has a broken alias.

InstallESD has a single folder inside of it: Packages.

 

[f54da]

>I do seem to end up with the same values of K between the shell and objective-C code, as best as I can tell.

Well yes because bswap will only practically affect the CID value passed in the request, not the value of K. Unless the objective-c version fails as well, I don't see how there can be any issues.

 

[Wowfunhappy]

Well yes because bswap will only practically affect the CID value passed in the request, not the value of K. Unless the objective-c version fails as well, I don't see how there can be any issues.

But isn't K generated via the bswapped CID value (authInfo.random)?

In order to test that the implementations are identical, I need to control all of the inputs. That means pre-setting authInfo.random like this:

//authInfo.random = generateRandomValue();
authInfo.random = CFSwapInt64(0x7C236FA375BF581E);

...instead of using the generateRandomValue function, because that would by definition be random.

Then I would use that same HEX for the CID in the shell script, and see if the K is the same. That's what I tried to do here and I got the same K value.

The CID will be the same because I'm setting it myself, right? How can I not do that and still have the function be deterministic?

 

[Jazzzny]

InstallESD itself contains a hidden BaseSystem.dmg - it is usable? It seems to be complete without packages, just like the RecoveryImage.

 

[f54da]

>But isn't K generated via the bswapped CID value (authInfo.random)?

I think of it as authInfo.random being mixed into K then CID = bswap(authInfo.random). But yes since byteswapping is an involution you can say that K is derived from byteswapped CID.

I'm guessing you tried this and it didn't work? Would be helpful if you had an example of a CID value which failed, then you could plug it into the objc-version and see what you get. There may be some constraint on CID which I wasn't seeing...

 

[Wowfunhappy]

InstallESD itself contains a hidden BaseSystem.dmg - it is usable? It seems to be complete without packages, just like the RecoveryImage.

...Turns out the hidden one inside InstallESD is bit identical to the one from the RecoveryImage, so I'm going to go with yes, it's usable. Great find!

This saves us a second download, but I think I still need to manually put the image together with hdiutil... right? Am I missing something? Why is the hidden DMG even in there?

Edit: Also this is is probably going to be the canonical thread that people stumble upon when looking for info on internet recovery inner workings, worth linking to https://www.afp548.com/2012/05/30/understanding-installesd-recovery-hd-internet-recovery/ as well

From the link:

InstallESD.dmg is the new-age Mac OS X retail DVD. As a Mac admin, you can use it to create an external bootable Lion install disk, build a NetInstall or NetRestore set, have it act as the Install DVD for an InstaDMG workflow, and more. A consumer installing Lion would have several pre-installation steps handled by the Install Mac OS X Lion application. After those completed, the installer would prepare InstallESD.dmg to be mounted as the boot volume. The computer would then restart and proceed with and finish the installation.

I don't think this is accurate at all for Mavericks. At minimum, we can quite trivially see that InstallESD.dmg is not usable on its own.

 

[Wowfunhappy]

I'm guessing you tried this and it didn't work? Would be helpful if you had an example of a CID value which failed, then you could plug it into the objc-version and see what you get. There may be some constraint on CID which I wasn't seeing...

Just so you know I'm not ignoring you —but I need to stop for the night. I tried playing with this again and I can't tell what's what anymore. (I can't seem to make Objective-C and shell generate the same K values no matter what, but I know that worked before so I'm clearly doing something wrong...)

What I can say for sure is that it doesn't fail on specific CIDs, I tried controlling for that. The same CID that does not work sometimes will work a different time. But it's possible it's a combination of the session token and CID that sometimes works and sometimes does not.

 

[f54da]

How strange... if it's not specific for a CID then maybe it's some form of ratelimiting? Not too sure. Looked through the disassembly again and can't find any evidence CID is anything other than pure randomness, maybe @dhinakg has some suggestion since he independently reveng'd the same thing.

I suppose you can always do the lazy thing and put it in a loop, even if the success probability is only 0.25 then just loop it 5 times to get 75% success prob.

 

[Wowfunhappy]

I suppose you can always do the lazy thing and put it in a loop, even if the success probability is only 0.25 then just loop it 5 times to get 75% success prob.

Yeah, I would have done that already except I don't want to overuse the donor information.

 

[dhinakg]

How strange... if it's not specific for a CID then maybe it's some form of ratelimiting? Not too sure. Looked through the disassembly again and can't find any evidence CID is anything other than pure randomness, maybe @dhinakg has some suggestion since he independently reveng'd the same thing.

My code is pretty half-baked at the moment so I currently just have a hardcoded CID (A64F96125D285330) stolen from a run of the actual Objective-C code. That has been working consistently for me, but at low volume (I've been running my code multiple times in a day, but not in rapid succession). I can poke around the script you posted and compare it to mine later - about to enter a very busy week.

Also remind me again what the difference is between InstallESD & BaseSystem?

BaseSystem = macOS recovery = RecoveryImage. It is basically "just enough" macOS to be able to reinstall it, or change startup security, or manipulate your disk, etc. InstallESD contains the packages that install macOS. You can boot the BaseSystem in various ways (Recovery partition on your computer, Internet Recovery, install USB) and get to Install macOS XYZ.app, but from there you will need to get the files to install macOS somehow (whether that is already on the install USB, or downloaded from the Internet).

 

[dhinakg]

What I can say for sure is that it doesn't fail on specific CIDs, I tried controlling for that. The same CID that does not work sometimes will work a different time. But it's possible it's a combination of the session token and CID that sometimes works and sometimes does not.

When it fails, what's the HTTP status?

 

[Wowfunhappy]

When it fails, what's the HTTP status?

401.

 

[dhinakg]

401.

Ok, so we know 401 likely means k doesn't match the expected value. Maybe there's an issue with the hashing that doesn't pop up all the time? I'll try to compare the values between my code, the actual code, @f54da's Objective-C and your script when I can.

 

[Wowfunhappy]

Alright, everything is fixed and we have donor values from a broken Mac courtesy of dosdude1.

curl https://mavericks.wowfunhappy.workers.dev | sh

In PMs, @Jazzzny figured out that the random errors were due to a combination of my hex_to_bin function mangling certain hex sequences, and the auth_info variable failing to properly store certain binary data.

On my end, I also removed some error checking in favor of enabling -e, which is probably safer overall and makes the code more readable.

I added the code to a github gist, then set up a Cloudflare worker to serve it with legacy http(s) compatibility. While it can be run via piping to sh, I do encourage reading the code first if you feel so inclined.

Thank you again to everyone who helped with this!!!

And, feel free to continue critiquing:

#!/bin/sh

# Download script written by Wowfunhappy.
# Thank you to Krackers, Jazzzny, and others for helping analyze Apple's download process and debug this script.
# Thank you to dosdude1 for donating identifiers from a broken Mac.
# Any mistakes are mine alone.

BOARD_SERIAL_NUMBER="C0243070168G3M91F"
BOARD_ID="Mac-3CBD00234E554E41"
ROM="003EE1E6AC14"

set -e

hex_to_bin() {
    echo -n "$1" | xxd -r -p
}

SERVER_ID=$(curl -fs -c - http://osrecovery.apple.com/ | tail -1 | awk '{print $NF}')
CLIENT_ID=$(dd if=/dev/urandom bs=8 count=1 2>/dev/null | od -An -tx1 | tr -d ' \n' | tr '[:lower:]' '[:upper:]')

# Generate K based on the client ID, server ID, ROM, board serial number, and board ID.
{
    hex_to_bin "$CLIENT_ID"
    hex_to_bin "$(echo $SERVER_ID | awk -F'~' '{print $2}')"
    hex_to_bin "$ROM"
    printf "%s" "${BOARD_SERIAL_NUMBER}${BOARD_ID}" | iconv -t utf-8 | openssl dgst -sha256 -binary
    printf '\xCC\xCC\xCC\xCC\xCC\xCC\xCC\xCC\xCC\xCC'
} > auth_info
# Convert auth_info to an sha256 hash. OpenSSL outputs raw binary which is then converted to uppercase hex.
K=$(openssl dgst -sha256 -binary < auth_info | od -An -tx1 | tr -d ' \n' | tr '[:lower:]' '[:upper:]')
rm auth_info

INSTALL_ESD_INFO=$(curl -fs 'http://osrecovery.apple.com/InstallationPayload/OSInstaller' -X POST \
-H 'Content-Type: text/plain' \
--cookie "session=$SERVER_ID" \
-d "cid=$CLIENT_ID
sn=$BOARD_SERIAL_NUMBER
bid=$BOARD_ID
k=$K

")

INSTALL_ESD_URL=$(echo "$INSTALL_ESD_INFO" | grep AU | awk -F': ' '{print $2}')
INSTALL_ESD_ASSET_TOKEN=$(echo "$INSTALL_ESD_INFO" | grep AT | awk -F': ' '{print $2}')

if [ "$INSTALL_ESD_URL" != "http://oscdn.apple.com/content/downloads/33/62/031-10295/gho4r94w66f5v4ujm0sz7k1m0hua68i6oo/OSInstaller/InstallESD.dmg" ]
then
    echo "Error: Server did not provide the Mavericks InstallESD URL." 1>&2
    exit 1
fi

echo "Downloading InstallESD.dmg..."
curl "$INSTALL_ESD_URL" -H "Cookie: AssetToken=$INSTALL_ESD_ASSET_TOKEN" > InstallESD.dmg

# Because we downloaded over unencrypted HTTP, it is critical that we verify the checksum.
echo "Verifying file integrity..."
if [ $(openssl dgst -sha256 InstallESD.dmg | awk -F'= ' '{print $2}') != "c861fd59e82bf777496809a0d2a9b58f66691ee56738031f55874a3fe1d7c3ff" ]
then
    rm InstallESD.dmg
    echo "Error: Download failed (mismatched checksum)" 1>&2
    exit 1
fi

if [ "$(uname)" = "Darwin" ]
then
    echo "Building InstallMacOSXMavericks.dmg..."
 
    # Ensure no volumes with these names are already mounted.
    hdiutil detach "/Volumes/OS X Base System" || true
    hdiutil detach "/Volumes/OS X Install ESD" || true
 
    hdiutil convert -ov "InstallESD.dmg" -format UDSP -o "InstallESD.sparseimage"
    hdiutil attach InstallESD.sparseimage -nobrowse
    hdiutil convert -ov "/Volumes/OS X Install ESD/BaseSystem.dmg" -format UDSP -o "BaseSystem.sparseimage"
    hdiutil resize -size 6056660992 "BaseSystem.sparseimage"
    hdiutil attach BaseSystem.sparseimage -nobrowse

    rm -rf "/Volumes/OS X Base System/System/Installation/Packages"
    cp -R "/Volumes/OS X Install ESD/Packages" "/Volumes/OS X Base System/System/Installation/"

    hdiutil detach "/Volumes/OS X Base System"
    hdiutil detach "/Volumes/OS X Install ESD"
 
    # Pause before converting so hdiutil won't fail with `hdiutil: convert failed - Resource temporarily unavailable`
    sleep 2
    hdiutil convert -ov "BaseSystem.sparseimage" -format UDZO -o "InstallMacOSXMavericks.dmg"
 
    rm InstallESD.sparseimage BaseSystem.sparseimage
    rm InstallESD.dmg
 
    echo "Successfully created InstallMacOSXMavericks.dmg"
 
else
    # The user will need to create the final installable image manually.
    echo "InstallESD.dmg successfully downloaded."
fi

Outstanding items:

• I still don't know why I don't have to (and in fact can't) do a byteswap.
• I still wish I could make the final image always have the exact same checksum.
• It would be nice to have an alternate code path for Linux and maybe other UNIX systems.
• I tested the script on Mavericks and Snow Leopard. Please feel free to test both ludicrously old systems (Tiger? Cheetah?) and new systems, as well as other UNIXs. The goal is to work everywhere.

Edit: The script does not work on Tiger because openssh does not support the sha256 algorithm. Oh well, I don't see a way around that! I suppose you could install modern openssh via MacPorts.

 

[Jazzzny]

One final thing - I think the downloaded files can be turned back into an "official App Store" installer application!

Mount the BaseSystem, copy the application to somewhere writable, and inside the application contents, create a folder named SharedSupport.

Then, inside the SharedSupport folder, copy in the InstallESD.dmg image, and OSInstall.mpkg (which can be obtained by making a request to
http://osrecovery.apple.com/InstallationPayload/Distribution in the same manner as http://osrecovery.apple.com/InstallationPayload/OSInstaller).

After doing this, running the installer prompts me to choose a disk and install as if it were an official installer.

If someone can verify that this installer functions normally (i.e. createinstallmedia and installing through the UI both work), then I think it would be a great addition to @Wowfunhappy 's script.

 

[Wowfunhappy]

After doing this, running the installer prompts me to choose a disk and install as if it were an official installer.

Does this not work if you just run the app from inside the DMG output by the script? Like you, I got up to choosing a disk, but I didn't run the UI all the way through or test createinstallmedia.

For simplicity, I would prefer that the script only output a single disk image, but it would be great if that disk image could be used as flexibly as possible.

 

[Jazzzny]

Does this not work if you just run the app from inside the DMG output by the script? Like you, I got up to choosing a disk, but I didn't run the UI all the way through or test createinstallmedia.

The installer from the disk image throws up a error stating "This is not a supported method of installing the operating system". I think having a 2-option menu (download installer application, download installer disk image) would still keep the script simple, but of course its your choice.

 

[Wowfunhappy]

The installer from the disk image throws up a error stating "This is not a supported method of installing the operating system".

Odd—that shows up for you before you choose a disk? Why does it work for me?

 

[Wowfunhappy]

Well, createinstallmedia definitely does not work:

$ sudo /Volumes/OS\ X\ Base\ System/Install\ OS\ X\ Mavericks.app/Contents/Resources/createinstallmedia --volume /Volumes/ThumbDrive --applicationpath /Volumes/OS\ X\ Base\ System/Install\ OS\ X\ Mavericks.app

/Volumes/OS X Base System/Install OS X Mavericks.app does not appear to be a valid OS installer application.

It would be great if there was a way to fix that. Based on interactions I've had with users over PMs and email, I actually suspect more people are going to try to use createinstallmedia than the UI.

I wonder if it would be possible to copy InstallESD.dmg inside of SharedSupport instead of copying InstallESD's packages to /System/Installation/Packages, and still end up with an image that can be booted? (Doing both would be stupid, we'd end up with an image that is twice the size.)

 

[Wowfunhappy]

...if that's not possible, maybe we do need options. Because as I think about it, there really are two equally canonical versions of the Mavericks installer: the GUI app you'd get from the app store, and the bootable image.

Edit: Actually—I think it would be best to just have the script output both. So at the end of the script you'd end up with two files: InstallMacOSXMavericks.dmg and "Install OS X Mavericks.app". The user can delete whatever they don't need.

Edit2:

Then, inside the SharedSupport folder, copy in the InstallESD.dmg image, and OSInstall.mpkg (which can be obtained by making a request to
http://osrecovery.apple.com/InstallationPayload/Distribution in the same manner as http://osrecovery.apple.com/InstallationPayload/OSInstaller).

I wonder what the difference is between OSInstall.mpkg obtained from OSInstaller, and the one inside of InstallESD.dmg/Packages. The two files are not bit-identical, although they are roughly the same size (729kb and 728kb, respectively).

 

[Grumpus]

Well, createinstallmedia definitely does not work:

When I run strings on createinstallmedia, I see that it references both InstallESD.dmg and BaseSystem.dmg, so I'd guess that InstallESD.dmg needs to be in SharedSupport for createinstallmedia to work.

$ strings createinstallmedia | grep dmg
BaseSystem.dmg
root-dmg=file://%@/%@%@
InstallESD.dmg
container-dmg=%@ root-dmg=%@
file:///BaseSystem.dmg
Mount of outer dmg failed.
BaseSystem missing from the outer dmg, damaged installer image?
Couldn't mount dmg %s (error code %d)

In the recent past, I used a Mavericks ISO put up on archive.org by @MrMacintoshBlog. The Install OS X Mavericks.app on that ISO has a SharedSupport directory containing both InstallESD.dmg and OSInstall.mpkg, and I can report that createinstallmedia does work there, albeit very, very slowly. The layout looks like this:

 

[startergo]

Manual USB installer creation from InstallESD.dmg:
- Attach InstallESD.dmg:

hdiutil attach /Volumes/Untitled/macOS\ Install\ Data/InstallESD.dmg

- Attach BaseSystem.dmg

hdiutil attach /Volumes/OS\ X\ Install\ ESD/BaseSystem.dmg

- Get the disk name of the BaseSystem.dmg:

diskutil list virtual | grep "OS X Base System"
  1:                  Apple_HFS OS X Base System        2.0 GB     disk9s1

- Find the USB number with diskutil list
- Format the USB:

diskutil partitionDisk disk7 HFS+ USBSTICK 100%

- Get the details of the USBSTICK:

diskutil list
/dev/disk7 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *15.7 GB    disk7
   1:                        EFI EFI                     209.7 MB   disk7s1
   2:                  Apple_HFS USBSTICK                15.4 GB    disk7s2

- Restore "OS X Base System" onto Apple_HFS USBSTICK

asr restore --source /dev/disk9s1 --target /dev/rdisk7s2 --erase --noprompt --noverify

- Remove the symlink for Packages folder:

rm -r /Volumes/OS\ X\ Base\ System/System/Installation/Packages

- Copy the Packages folder to "OS X Base System" Volume inside /System/Installation/ subfolder:

cp -rpv /Volumes/OS\ X\ Install\ ESD/Packages /Volumes/OS\ X\ Base\ System/System/Installation/Packages

- Copy BaseSystem.chunklist and BaseSystem.dmg to the root of the USB installer:

cp /Volumes/OS\ X\ Install\ ESD/BaseSystem.chunklist "/Volumes/OS X Base System"
cp /Volumes/OS\ X\ Install\ ESD/BaseSystem.dmg "/Volumes/OS X Base System"

- Rename the USB:

diskutil rename OS\ X\ Base\ System Install\ Mavericks
sudo bless --folder /Volumes/Install\ Mavericks/System/Library/CoreServices --label Install\ Mavericks
cp -v /Volumes/Install\ Mavericks/Install\ OS\ X\ Mavericks.app/Contents/Resources/InstallAssistant.icns /Volumes/Install\ Mavericks/.VolumeIcon.icns

- Detach the mounted images:

hdiutil detach /Volumes/OS\ X\ Install\ ESD
hdiutil detach /Volumes/OS\ X\ Base\ System

 

[Jazzzny]

I wonder what the difference is between OSInstall.mpkg obtained from OSInstaller, and the one inside of InstallESD.dmg/Packages. The two files are not bit-identical, although they are roughly the same size (729kb and 728kb, respectively).

The .mpkg files are actually xar files and can be extracted with xar -xf [file.mpkg].

I believe the file of interest is the "Distribution". When the Distribution from the IR OSInstall is diffed with the dmg OSInstall, it seems to be a slightly updated version.

 

[Wowfunhappy]

I updated the script to copy BaseSystem.chunklist and BaseSystem.dmg into the root of the final image. According to an email I received, this is necessary for the installer to create a recovery partition thus for things FireVault to work.