Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Can we download Mavericks directly from Apple's servers?
- Thread starter Wowfunhappy
- Start date
- Sort by reaction score
Oops, I read your Objective-C code again and realized that bswap is only used when you're generating the random CID.
I'd originally thought that the CID sent to the server, and the CID used to generate K, were bswapped versions of each other.
But then I don't understand, how is the bswap significant from the server's perspective? The CID is just randomness, isn't it?
-----
I'm pretty convinced we're going to need a real board serial number and rom combination, or at least that if it's possible to generate synthetic values, they would need to have some property which the Hackintosh community has not yet reverse engineered.
I tried many combinations from GenSMBIOS.py and none of them ever worked, whereas the real values from my Macbook Air's work about half of the time. (I don't understand why the real values don't work all the time, but oh well.)
As an aside, I discovered my 2014 Macbook Air—a "MacBookAir6,1" in more technical parlance—can actually run Mountain Lion according to https://dortania.github.io/OpenCore-Install-Guide/extras/smbios-support.html#macbook-air. Since we know it works, perhaps this opens up the number of potential donor macs.
Last edited:
maybe it helps, listing the data points a Mac makes unique. All are in the firmware. Read with the Dumper (the test_nvram cli tool lists all, when called via the Dumper it hides some, preventing people posting screenshots).
Here now on a MBA62, too
data points made unusable, of course
So there is
in Fsys:
ssn (system serial number)
hwc (hardware code, usually the end of ssn)
son (sales order number, for some machines a generic code, for some unique)
in BootLoader:
BD (build date)
LBSN (logic board serial number)
Firmware MAC
As when rebuilding Mac Pro firmwares we know, if there are some of them botched, some Apple services fail.
Here now on a MBA62, too
data points made unusable, of course
Code:
% test_nvram -dump
no spi-flash type given, but found MBA62. Assuming N25Q064..3E from experience
...
===================================================
test_nvram_shell_script 13-Dec-2024 by Macschrauber
scanning: /Users/user/Downloads/14.12.2024_18-39-05_N25Q064..3E_dump.bin
...
MBA62, serial from firmware: C17xxxxxxxG085
LBSN: C024174xxxxxxY5CAE BD: 1404xx1404xx Firmware MAC: 0c:4d:xx:xx:xx:ea
hwc: G085 son: MD760D/
Fsys: 0 overrides, 1 override-version, 2 kcal, 3 ssn, 4 hwc, 5 son, 6 EOF
...So there is
in Fsys:
ssn (system serial number)
hwc (hardware code, usually the end of ssn)
son (sales order number, for some machines a generic code, for some unique)
in BootLoader:
BD (build date)
LBSN (logic board serial number)
Firmware MAC
As when rebuilding Mac Pro firmwares we know, if there are some of them botched, some Apple services fail.
Thanks, however we know for sure that other data cannot be related because it isn't used in the request to Apple.
The only data sent to Apple's servers is:
- The session token (this is sent to us from Apple's servers upon request. When we request the session token, we don't send any data to Apple.)
- The board serial number (also sometimes called the "MLB".)
- The board id (not unique to a given Mac)
- The "K"
- The session token (same as above)
- The CID (a randomly generated 16 character hexadecimal sequence)
- The ROM
- The board serial number (same as above)
- The board id (same as above)
(The one minor mystery here is the CID—we don't know what it's for, and I suppose it's theoretically possible there is more information encoded inside of it. But it really looks like Apple's code is using random generation, and our requests with randomly generated CIDs often work, and the requests which do fail don't consistently fail with some CIDs and not others.)
So, the only computer-specific information ever used in the request to Apple is the board serial number and the rom. Apple cannot see anything else!
Last edited:
just to get you correctly, this is the LBSN stored in the BootLoader
what can be read by
Code:
nvram_mlb="$(nvram 4D1EDE05-38C7-4A6A-9CC6-4BCCA8B38C14:MLB)"; nvram_mlb="${nvram_mlb##*$'\t'}"btw: those 2 variables
4D1EDE05-38C7-4A6A-9CC6-4BCCA8B38C14:MLB and 4D1EDE05-38C7-4A6A-9CC6-4BCCA8B38C14:ROM are not stored inside the NVRAM physically.Seems the OS maps them.
As I dont find it in a dump, read from the spi flash chip.
Coincidences are funny. I started looking into Internet Recovery again a little while ago for other purposes, and then @Jazzzny mentioned a few days ago how you can download Mavericks through the servers, which answered a few questions I had (but not others).
Anyways, I did my own reverse engineering of the protocol, and I am fairly certain that it is impossible to get it to work without valid values from a real Mac. As mentioned before, the only values sent to Apple are:
So let's see what Apple needs in order to compute and verify the hash.
Here, we go back to the initial point about the hash being non-reversable. Apple must know the ROM ahead of time because it is never passed in plaintext to the server. The MLB, however, is passed in plaintext to the server. Apple uses the MLB to find the ROM, inserts the values into our formula, and then validates against what we pass in.
So our two options here are:
This does not prove or disprove whether the ROM is derived from the MLB; it solely tells us that the MLB is looked up in a database. I personally think the ROM is fetched from the database, rather than derived from the serial (otherwise, why bother with the ROM anyway? Additionally, if it was derivable, why didn't Apple use it for validation for iMessage registration?), but in any case, we still need a valid MLB that is in Apple's database. At that point, just get the ROM from that real Mac as well - trying to find a correlation between ROM and MLB would require a large collection of both and is probably not worth the effort.
Anyways, I did my own reverse engineering of the protocol, and I am fairly certain that it is impossible to get it to work without valid values from a real Mac. As mentioned before, the only values sent to Apple are:
cid: client session ID, randomly generated (likely for anti-replay purposes)sn: the MLB, uniquely identifyingbid: the board IDk
k is the hash ofcidsid: server session ID (likely included here for anti-replay purposes)- ROM
- the hash of
- MLB
- board ID
CCCCCCCCCCCCCCCCCCCC
k will fail the response, so Apple is validating the value somehow. The only way Apple could be doing this is if they are computing k themselves. This hash cannot be precomputed, as it covers cid, which we know is randomly generated, so this must be occurring at request time.So let's see what Apple needs in order to compute and verify the hash.
cid is random so it must have no bearing on the validation. sid is provided by the servers so it can't have any impact on deciding whether a machine is real or not either. Likewise, the CCCCCCCCCCCCCCCCCCCC constant is a constant, so that leaves the ROM and the hash of MLB and the board ID.Here, we go back to the initial point about the hash being non-reversable. Apple must know the ROM ahead of time because it is never passed in plaintext to the server. The MLB, however, is passed in plaintext to the server. Apple uses the MLB to find the ROM, inserts the values into our formula, and then validates against what we pass in.
So our two options here are:
- The ROM can be derived from the MLB (but to this day, no one has found a correlation)
- Apple is looking up the ROM based on the MLB given.
k does not match the computed value.) But if I swap the serial to something generated from macserial, I get a 403 back. The MLB format is fairly well known at this point, so I doubt that it's an issue with the serials themselves. So, there must be some kind of database lookup before the ROM validation step, and if we peer into the binary, there is a codepath for 403 that results in logging "Machine not in database, falling back to app store..."This does not prove or disprove whether the ROM is derived from the MLB; it solely tells us that the MLB is looked up in a database. I personally think the ROM is fetched from the database, rather than derived from the serial (otherwise, why bother with the ROM anyway? Additionally, if it was derivable, why didn't Apple use it for validation for iMessage registration?), but in any case, we still need a valid MLB that is in Apple's database. At that point, just get the ROM from that real Mac as well - trying to find a correlation between ROM and MLB would require a large collection of both and is probably not worth the effort.
The question is let's assume there are real values. The moment they get out in the wild they will be used to make iCloud work and multiple requests will most definitely make Apple block them. So these identifiers should be obfuscated to the general public.
>ROM is fetched from the database, rather than derived from the serial
The other hypothesis I have is that maybe it's some sort of TOFU (trust on first use) kind of thing, where instead of having a lookup table for all factory serial numbers, the first time the machine phones home to apple it gets recorded somewhere. It'd be easy to test this by using the values from a hackintosh that has been actually set up properly and has logged into apple id at least once.
This also does remind me that when mac computers are first booted, there is a page to accept the warranty information which is never seen again on reinstalls. I've always wondered where exactly that information is recorded.
The other hypothesis I have is that maybe it's some sort of TOFU (trust on first use) kind of thing, where instead of having a lookup table for all factory serial numbers, the first time the machine phones home to apple it gets recorded somewhere. It'd be easy to test this by using the values from a hackintosh that has been actually set up properly and has logged into apple id at least once.
This also does remind me that when mac computers are first booted, there is a page to accept the warranty information which is never seen again on reinstalls. I've always wondered where exactly that information is recorded.
Worth mentioning is that Fusion creates VM's that can log in to iCloud using the same MLB and ROM, but different HID and serial and FW version. VMware generates a serial number accepted by iCloud. I used the same info to create a VMware Workstation without passing the MLB and iCloud was still working.
Last edited:
Why didn't Apple use it for validation for iMessage registration regardless? Like, whether it's derived or looked up in a database, if that database exists Apple could check it before allowing iMessage registration.
(By the way, in addition to everything else, thank you for explaining what "CID" stands for, that makes sense now!)
I've also had a Hackintosh experience that would support this. Apple's website says a serial number which is definitely not real is real, and that the computer was purchased on the same day as when I first booted that Hackintosh.
However, this is only the case for one Hackintosh system. The system I use right now, for example, does not have a valid serial number according to Apple. I can still use iMessage, however.
Last edited:
So @startergo very kindly sent me the MLB/ROM combination from a Macbook Pro 11,3 he had, in case that would work. Interestingly, when I plugged those values into my shell script, they worked, but the script downloaded Yosemite instead of Mavericks.
So this data is actually being used by Apple to determine the correct OS to send, it's not purely about validation.
So this data is actually being used by Apple to determine the correct OS to send, it's not purely about validation.
...what's even more strange is that the Macbook Pro 11,3 was released in June 2014.
It 100% should have shipped with Mavericks. Yosemite did not exist yet. WTF?
It 100% should have shipped with Mavericks. Yosemite did not exist yet. WTF?
The initial OS for a MBP11,3 is OS X 10.9 (13A3017)
First assumption was: they send the original OS
First assumption was: they send the original OS
Why would they be used? The only reason I see to use the values is hackintoshing, and iMessage/FaceTime (iCloud and the rest work already) are already a solved problem there.
I doubt it, as serials in Check Coverage stay invalid even if a hackintosh is using them, but I'll find some values to check with.
What is even weirder is that board ID is used by macrecovery.py to download BigSur recovery partition:
Code:
Big Sur
./macrecovery.py -b Mac-2BD1B31983FE1663 -m 00000000000000000Just a note, as I said above—I have had one case where this was literally not true for me. But I recognize this is not a common experience.
Some configurations of the 11,3 were shipped until May 19, 2015, according to EveryMac. Thats a comfortable ~7 months after Yosemite was released, which could indicate that Apple updated their internal information (tied to MLB serial?) to start sending out the updated internet recovery for machines shipped after Yosemite was released. You could probably check this by finding the manufacture date of the machine using something like macserial.
Oops, looks like I missed some messages before I sent my last reply.
Fair point.
How is the serial not real? Unless you previously checked the state beforehand and it was invalid, or if the manufacturing date is after when Apple stopped producing the device, I see no reason why it couldn't be a coincidence. On the other hand, I am pretty sure that Apple Support can make previously invalid serials come up as valid.
The returned OS is known to change based on the MLB and is likely correlated to when it was manufactured.
The recovery endpoint has various quirks, one of which is that it will accept invalid MLBs, but return the latest OS for them. See macrecovery.
The serial was previously invalid. In addition, the purchase date was listed as the first day I installed Hackintosh on this particular computer. I suppose it's possible that could be a coincidence, but it's a pretty big one.
Alright... we're mostly done here!
This script will automatically:
Step #3 will only work if you're running macOS. It would eventually be neat to have an alternate code path for Linux (and maybe even other UNIX systems) which uses one or more of the utilities here.
Step #3 is also apparently not deterministic, the final
For Step #2, I elected not to use (most of) the real serial number, or generate a real K, since this server doesn't care and I figured it's best to use the donor information as little as possible. Please let me know if you disagree and think we should try to make the requests as real as we can.
Lastly, I still don't know why the initial request to the
Please also let me know if anything should be commented better. I want the script to be very easy to read so anyone can verify that Mavericks is being downloaded from official sources.
Bash:
#!/bin/sh
BOARD_SERIAL_NUMBER="C02XXXXXXXXXXXXXX"
BOARD_ID="Mac-35C1E88140C3E6CF"
ROM="XXXXXXXXXXXX"
hex_to_bin() {
printf "$(echo "$1" | sed 's/\(..\)/\\x\1/g')"
}
generate_random_hex() {
dd if=/dev/urandom bs=$(($1 / 2)) count=1 2>/dev/null | od -An -tx1 | tr -d ' \n' | tr '[:lower:]' '[:upper:]'
}
SERVER_ID=$(curl -fs -c - http://osrecovery.apple.com/ | tail -1 | awk '{print $NF}')
if [ -z $(echo "$SERVER_ID" | tr -d '[:space:]') ]
then
echo "Error: Server did not respond with a session token." 1>&2
exit 1
fi
CLIENT_ID=$(generate_random_hex 16)
# Generate `K` based on the client ID, server ID, ROM, board serial number, and board ID.
auth_info=$(printf '%b%b%b%b%b' \
$(hex_to_bin "$CLIENT_ID") \
$(hex_to_bin $(echo $SERVER_ID | awk -F'~' '{print $2}')) \
$(hex_to_bin "$ROM") \
$(printf "%s" "${BOARD_SERIAL_NUMBER}${BOARD_ID}" | openssl dgst -sha256 -binary) \
'\xCC\xCC\xCC\xCC\xCC\xCC\xCC\xCC\xCC\xCC')
# Convert auth_info to an sha256 hash. OpenSSL outputs raw binary which is then converted to uppercase hex.
K=$(printf "%s" "$auth_info" | openssl dgst -sha256 -binary | od -An -tx1 | tr -d ' \n' | tr '[:lower:]' '[:upper:]')
INSTALL_ESD_INFO=$(curl -fs 'http://osrecovery.apple.com/InstallationPayload/OSInstaller' -X POST \
-H 'Content-Type: text/plain' \
--cookie "session=$SERVER_ID" \
-d "cid=$CLIENT_ID
sn=$BOARD_SERIAL_NUMBER
bid=$BOARD_ID
k=$K
")
if [ -z $(echo "$INSTALL_ESD_INFO" | tr -d '[:space:]') ]
then
echo "Error: Server did not respond with InstallESD information." 1>&2
exit 1
fi
INSTALL_ESD_URL=$(echo "$INSTALL_ESD_INFO" | grep AU | awk -F': ' '{print $2}')
INSTALL_ESD_ASSET_TOKEN=$(echo "$INSTALL_ESD_INFO" | grep AT | awk -F': ' '{print $2}')
if [ "$INSTALL_ESD_URL" != "http://oscdn.apple.com/content/downloads/33/62/031-10295/gho4r94w66f5v4ujm0sz7k1m0hua68i6oo/OSInstaller/InstallESD.dmg" ]
then
echo "Error: Server did not provide the Mavericks InstallESD URL." 1>&2
exit 1
fi
curl "$INSTALL_ESD_URL" -H "Cookie: AssetToken=$INSTALL_ESD_ASSET_TOKEN" > InstallESD.dmg
# Because we downloaded over unencrypted HTTP, it is critical that we verify the checksum.
echo "Verifying file integrity..."
if [ $(openssl dgst -sha256 InstallESD.dmg | awk -F'= ' '{print $2}') != "c861fd59e82bf777496809a0d2a9b58f66691ee56738031f55874a3fe1d7c3ff" ]
then
rm InstallESD.dmg
echo "Error: Download failed (mismatched checksum)" 1>&2
exit 1
fi
# We successfully downloaded InstallESD. Next, download BaseSystem.dmg.
# For this request, K can be random. The Board Serial Number can be random except for certain characters.
# Refresh Server ID, which may be stale by this point.
SERVER_ID=$(curl -fs -c - http://osrecovery.apple.com/ | tail -1 | awk '{print $NF}')
BASE_SYSTEM_INFO=$(curl -fs 'http://osrecovery.apple.com/InstallationPayload/RecoveryImage' -X POST \
-H 'Content-Type: text/plain' \
-H 'User-Agent: InternetRecovery/1.0' \
--cookie "session=$SERVER_ID" \
-d "cid=$(generate_random_hex 16)
sn=C02$(generate_random_hex 8)$(echo "$BOARD_SERIAL_NUMBER" | cut -c12-15)$(generate_random_hex 2)
bid=$BOARD_ID
k=$(generate_random_hex 64)
os=default
fg=$(generate_random_hex 64)
")
if [ -z $(echo "$BASE_SYSTEM_INFO" | tr -d '[:space:]') ]
then
echo "Error: Server did not respond with RecoveryImage information." 1>&2
exit 1
fi
BASE_SYSTEM_URL=$(echo "$BASE_SYSTEM_INFO" | grep AU | awk -F': ' '{print $2}')
BASE_SYSTEM_TOKEN=$(echo "$BASE_SYSTEM_INFO" | grep AT | awk -F': ' '{print $2}')
if [ "$BASE_SYSTEM_URL" != "http://oscdn.apple.com/content/downloads/33/62/031-10295/gho4r94w66f5v4ujm0sz7k1m0hua68i6oo/RecoveryImage/RecoveryImage.dmg" ]
then
echo "Error: Server did not provide the Mavericks BaseSystem URL." 1>&2
exit 1
fi
curl "$BASE_SYSTEM_URL" -H "Cookie: AssetToken=$BASE_SYSTEM_TOKEN" > RecoveryImage.dmg
# Because we downloaded over unencrypted HTTP, it is critical that we verify the checksum.
echo "Verifying file integrity..."
if [ $(openssl dgst -sha256 BaseSystem.dmg | awk -F'= ' '{print $2}') != "4e33ed34db869189bec60dc2334573e5d5e420037fc18e7f0f4b1251a396ad6d" ]
then
rm BaseSystem.dmg
echo "Error: Download failed (mismatched checksum)" 1>&2
exit 1
fi
if [ "$(uname)" = "Darwin" ]
then
echo "Building InstallMacOSXMavericks.dmg..."
hdiutil convert "InstallESD.dmg" -format UDSP -o "InstallESD.sparseimage"
hdiutil convert "BaseSystem.dmg" -format UDSP -o "BaseSystem.sparseimage"
hdiutil resize -size 6056660992 "BaseSystem.sparseimage"
hdi_util_output=$(hdiutil attach BaseSystem.sparseimage -nobrowse)
if ! echo "$hdi_util_output" | grep "/Volumes/OS X Base System"
then
echo "Error mounting BaseSystem" 1>&2
exit 1
fi
hdi_util_output=$(hdiutil attach InstallESD.sparseimage -nobrowse)
if ! echo "$hdi_util_output" | grep "/Volumes/OS X Install ESD"
then
echo "Error mounting InstallESD" 1>&2
exit 1
fi
rm -rf "/Volumes/OS X Base System/System/Installation/Packages"
cp -R "/Volumes/OS X Install ESD/Packages" "/Volumes/OS X Base System/System/Installation/"
hdiutil detach "/Volumes/OS X Base System"
hdiutil detach "/Volumes/OS X Install ESD"
# Pause before converting so hdiutil won't fail with `hdiutil: convert failed - Resource temporarily unavailable`
sleep 2
hdiutil convert "BaseSystem.sparseimage" -format UDZO -o "InstallMacOSXMavericks.dmg"
rm InstallESD.sparseimage BaseSystem.sparseimage
echo "Successfully created InstallMacOSXMavericks.dmg!"
else
# The user will need to create the final installable image manually.
echo "Files successfully downloaded."
fiThis script will automatically:
- Download InstallESD (as before)
- Download RecoveryImage.dmg
- Use #1 and #2 to build an installable InstallMacOSXMavericks.dmg image.
Step #3 will only work if you're running macOS. It would eventually be neat to have an alternate code path for Linux (and maybe even other UNIX systems) which uses one or more of the utilities here.
Step #3 is also apparently not deterministic, the final
InstallMacOSXMavericks.dmg has a different checksum each time. This makes me a bit sad, it would be great if we could ensure everyone was getting an identical, perfect copy of Mavericks at the end.For Step #2, I elected not to use (most of) the real serial number, or generate a real K, since this server doesn't care and I figured it's best to use the donor information as little as possible. Please let me know if you disagree and think we should try to make the requests as real as we can.
Lastly, I still don't know why the initial request to the
OSInstaller endpoint only works sometimes!!!Please also let me know if anything should be commented better. I want the script to be very easy to read so anyone can verify that Mavericks is being downloaded from official sources.
Last edited:
^ I still don't see a bswap being done between CID & cid-in-K anywhere. Does this work reliably?
Also remind me again what the difference is between InstallESD & BaseSystem? I thought InstallESD had the "install osx.app" which had BaseSystem embedded inside of it (last time I analyzed this I remember that it did something clever where it uses an obscure kernel flag to effectively mount and chroot the core system libs). But it seems like you're doing the reverse, putting InstallESD inside base system?
Edit: Also this is is probably going to be the canonical thread that people stumble upon when looking for info on internet recovery inner workings, worth linking to https://www.afp548.com/2012/05/30/understanding-installesd-recovery-hd-internet-recovery/ as well
Also remind me again what the difference is between InstallESD & BaseSystem? I thought InstallESD had the "install osx.app" which had BaseSystem embedded inside of it (last time I analyzed this I remember that it did something clever where it uses an obscure kernel flag to effectively mount and chroot the core system libs). But it seems like you're doing the reverse, putting InstallESD inside base system?
Edit: Also this is is probably going to be the canonical thread that people stumble upon when looking for info on internet recovery inner workings, worth linking to https://www.afp548.com/2012/05/30/understanding-installesd-recovery-hd-internet-recovery/ as well