Other A proxy to fix HTTPS issues on old versions of OS X

[maverick28]

Apple Mail refuses to connect to iCloud IMAP servers. Lots of messages in the Console of the type:

Mail[172]: CFNetwork SSLHandshake failed (-9836)

I've been seeing this since yesterday. It looks like this error began to slip through Squid.
Interestingly, I received my message sent from iOS Apple Mail to one of these iCloud accounts; however, Apple's notice messages sent to the "iCloud 2" account went missing. Upon logging into the iCloud on the Web, I was greeted by the TOC acceptance prompt, which I agreed to in both iCloud accounts. This has made no difference to Apple Mail on Mavericks.

That "iCloud 2" account uses the legacy auth scheme - no 2FA or 2S. I'm puzzled as to whether this has smth to do with my not receiving that Apple notice message. Be that as it may, it shows IMAP offline for both accounts.

 

[iPodNano3]

Apple Mail refuses to connect to iCloud IMAP servers. Lots of messages in the Console of the type:

Mail[172]: CFNetwork SSLHandshake failed (-9836)

I've been seeing this since yesterday. It looks like this error began to slip through Squid.
Interestingly, I received my message sent from iOS Apple Mail to one of these iCloud accounts; however, Apple's notice messages sent to the "iCloud 2" account went missing. Upon logging into the iCloud on the Web, I was greeted by the TOC acceptance prompt, which I agreed to in both iCloud accounts. This has made no difference to Apple Mail on Mavericks.

That "iCloud 2" account uses the legacy auth scheme - no 2FA or 2S. I'm puzzled as to whether this has smth to do with my not receiving that Apple notice message. Be that as it may, it shows IMAP offline for both accounts.

View attachment 2523348

try manually adding the account:

iCloud Mail server settings for other email client apps - Apple Support

Use these iCloud Mail server settings to manually set up your iCloud Mail account in an email client app.

support.apple.com

this may not work on mavericks and it didnt work for me because the mail app recognized it was an icloud mail server and proceeded to continue with the icloud account type

i remember having the same issue with my icloud mail on mavericks for a while, about two weeks or so and it just randomly fixed itself

 

[DurltazorOSXPower]

Hello guys here I'm also taking the same problem in snow leopard, the IMAP server refuses to connect.

 

[iPodNano3]

update: it broke for me again, perhaps it was always broken but mail just never synced the account, so it didnt know it was broken

 

[Wowfunhappy]

I'll have a beta of the new proxy ready relatively soon. I'm not 100% sure if it will fix IMAP—(technically that's not an http connection, it's imap—right?)—but it may.

 

[DurltazorOSXPower]

We cheer for this, strangeness seems that the server an IMAP has been turned off ..

The icloud.com IMAP server “automatically added to iOS and x mail” is not responding. Try to verify the network connection and confirm the server name is correct. If so, O server may be temporarily indisputable. If you continue, you may not get messages.

I may be talking ********, of course ..
But it is not connecting to any system before the OSX El Capitan.

 

[macnavarra]

We cheer for this, strangeness seems that the server an IMAP has been turned off ..

The icloud.com IMAP server “automatically added to iOS and x mail” is not responding. Try to verify the network connection and confirm the server name is correct. If so, O server may be temporarily indisputable. If you continue, you may not get messages.

I may be talking ********, of course ..
But it is not connecting to any system before the OSX El Capitan.

A Autograph! from the network administrator and strong Cheer could be the solution for the problem.

 

[f54da]

I think imap can use ssl which is probably what is now failing. imap can be done over HTTP proxies that support the connect method (allow shuffling arbitrary bytes through the tcp connection), which is how the golang-based proxy operates. Only issue is that most email clients probably don't support using http proxy.

You could probably write some IMAP specific proxy though, hardcoding the destination.

 

[f54da]

Btw @Wowfunhappy have you verified that proxy properly handles bad certs? I.e. it should not open you up to a malicious MITM. https://badssl.com/. Not sure how to handle the UX in such a case. Maybe terminate the inbound connection and log something. Or if you want to be fancy, return a known invalid cert with some error code embedded in the cert details.

 

[Wowfunhappy]

Btw @Wowfunhappy have you verified that proxy properly handles bad certs? I.e. it should not open you up to a malicious MITM. https://badssl.com/.

I have verified that, yes!

Not sure how to handle the UX in such a case. Maybe terminate the inbound connection and log something.

UX is bad. Yes, basically the connection fails and Go logs an error. IIRC this worked out of the box with Go, I didn't have to do anything special.

 

[Wowfunhappy]

...from what I understand, to fix IMAP we need a SOCKS proxy instead of an https proxy. That operates on a lower level.

Edit: Nope, using SOCKS does not make iCloud Mail work, even though it certainly looks like Apple Mail is sending traffic through the proxy.

 

[Wowfunhappy]

Alright, I have a way to fix iCloud Mail.

The way this is going to work is you'll manually add an IMAP account with the server set to localhost, and your username set to originalEmail@originalDomain@IMAPServerAddress.

So for iCloud, that would be foobar@icloud.com@imap.mail.me.com.

 

[f54da]

Neat, you (Claude) wrote an imap proxy. There's an existing one at https://github.com/emersion/go-imap-proxy I don't know if one is cleaner than the other. Probably the one you have is fine since it's zero dependencies.

 

[Wowfunhappy]

There's an existing one at https://github.com/emersion/go-imap-proxy

I was having some trouble getting that one to work, although it's probably my fault and I didn't try very hard.

By the way, I've settled on shipping five root certificates with the proxy:

1. ISRG Root X1
2. DigiCert Assured ID Root G2
3. DigiCert Assured ID Root G3
4. USERTrust RSA Certification Authority
5. COMODO ECC Certification Authority

The first four are the ones recommended by the LegacyJailbreak subreddit, and appear to be sufficient for basically every website. The fifth certificate is used by the modern Apple Maps API, which of course doesn't currently work on old systems anyway (I haven't been able to replicate iPodNano3's method in my proxy) but the fact that it's needed to access an Apple domain is enough reason to include it.

 

[GalacticStag]

Alright, I have a way to fix iCloud Mail.

The way this is going to work is you'll manually add an IMAP account with the server set to localhost, and your username set to originalEmail@originalDomain@IMAPServerAddress.

So for iCloud, that would be foobar@icloud.com@imap.mail.me.com.

i'm finding trouble setting up iCloud Mail with that method. Do i put localhost as my incoming mail server/outgoing mail server or on the imap route prefix?

 

[Wowfunhappy]

i'm finding trouble setting up iCloud Mail with that method. Do i put localhost as my incoming mail server/outgoing mail server or on the imap route prefix?

Sorry, I should have been clear, this won't work yet. You need my (new, forthcoming) proxy server. I will release something soon, I have been literally working on this full time for the past several days.

 

[f54da]

@Wowfunhappy you can speed up proxy by only falling back to slower mitm codepath if the TLS SNI matches the things on the redirect whitelist. Otherwise do the raw bytestream copy between client/server.

Also in https://github.com/Wowfunhappy/AquaProxy/commit/7abf73a1e13e89c73cf763775856fd9d91b54aa3 why do you need a mutex? Doesn't go have an immutableMap equivalent?

 

[Wowfunhappy]

@f54da I really appreciate your comments, the vast majority of new code in this project was essentially "vibe-coded" with Claude, and I'm not able to understand a lot of it. This makes me kind of uncomfortable, but it was the only way I could reasonably build the program myself, and it has let me add a bunch of features I really wanted, such as the ability to distinguish between "modern" and "legacy" clients.

@Wowfunhappy you can speed up proxy by only falling back to slower mitm codepath if the TLS SNI matches the things on the redirect whitelist. Otherwise do the raw bytestream copy between client/server.

...I thought it pretty much was doing that. Is it not?

In serveConnect:

    // Check if domain has redirect rules
    // Extract domain without port
    domain := host
    if h, _, err := net.SplitHostPort(host); err == nil {
        domain = h
    }
    
    redirectMutex.RLock()
    hasRedirects := redirectDomains[domain]
    redirectMutex.RUnlock()
    
    
    // Route based on client capabilities and redirect rules
    if clientHello.isModernClient && !hasRedirects && !*forceMITM {
        // Modern client detected, no redirects, and force MITM not enabled - use passthrough mode
        log.Printf("[%s] PASSTHROUGH: %s", connID, host)
        p.passthroughConnection(clientConn, host, clientHello, connID)
    } else {
        // Legacy client OR domain has redirects OR force MITM enabled - use MITM mode
        if *forceMITM {
            log.Printf("[%s] MITM (forced): %s", connID, host)
        } else if hasRedirects {
            log.Printf("[%s] MITM (redirects): %s", connID, host)
        } else {
            log.Printf("[%s] MITM (legacy): %s", connID, host)
        }
        p.serveMITM(clientConn, host, name, clientHello, connID)
    }

Requests that clearly came from a modern client and don't have redirects aren't MITM'd at all.

And then for requests that are MITM'd:

    // Check if domain has redirect rules
    // Extract domain without port
    domain := host
    if h, _, err := net.SplitHostPort(host); err == nil {
        domain = h
    }
    
    redirectMutex.RLock()
    hasRedirects := redirectDomains[domain]
    redirectMutex.RUnlock()
    
    // If URL logging is enabled OR domain has redirects, parse HTTP requests
    if *logURLs || hasRedirects {
        // Parse and handle HTTP requests
        p.handleMITMWithLogging(tlsConn, serverConn, host, connID, hasRedirects)
    } else {
        // Use efficient raw TCP/TLS forwarding
        done := make(chan bool, 2)
        
        // Client to server
        go func() {
            copyData(serverConn, tlsConn, connID, "Client→Server")
            // For TLS connections, we can't use half-close, but we avoid closing
            // the connection until both directions are done
            done <- true
        }()
        
        // Server to client
        go func() {
            copyData(tlsConn, serverConn, connID, "Server→Client")
            done <- true
        }()
        
        // Wait for both directions to complete
        <-done
        <-done
        
        // Now close both connections
        tlsConn.Close()
        serverConn.Close()
    }

Is there a way to make it even faster?

Also in https://github.com/Wowfunhappy/AquaProxy/commit/7abf73a1e13e89c73cf763775856fd9d91b54aa3 why do you need a mutex? Doesn't go have an immutableMap equivalent?

That I can't answer beyond "because the AI did it that way". However, I actually reverted this particular commit later on. I was trying to make Apple Maps work, but I had to give up, and without that I didn't see a need for custom headers.

 

[f54da]

>I thought it pretty much was doing that. Is it not?

Umm I guess. The whole thing seems kind of spaghetty-ish so maybe I mistraced it. Looking again, it does seem to be using the desired hostname in the initial HTTP CONNECT to branch between slow/fast codepaths (which is actually a better idea in retrospect than my suggestion of using the SNI in the handshake).

• serveConnect – serves initial HTTP CONNECT
  • Send 200 even before initiating outbound connection to indicated server, wait for client TLS hello
  • Get client capabilities from TLS Hello
• if client is modern AND initial hostname from the CONNECT was to non-redirect domain, behave like a "normal" HTTP CONNECT proxy (passthroughConnection)
  • Initiate raw tcp con to destination host, and proxy the raw bytes from client back/forth
• Otherwise proceed to MITM (serveMITM)
  • Split what would usually be a single tls session between client -> destination (proxied over client->proxy and proxy->destination tcp tunnels) into two TLS negotiations, one between client->proxy and proxy->destination
  • If there are redirects, it seems to reconnect to the redirected hostname. This seems weird, why not check for redirects before the initial outgoing dial in line 1355?
  • There's multiple lookups to redirectRules[domain] in there, why? You only need to do it once.
  • If no redirects, use a fast-path where underlying http stream between client->proxy and proxy->server are just direct byte copies
  • Otherwise use a codepath that actually parses the http payload from client->proxy so it can be modified

As for the mutex, looking again it has separate readlock/write lock, so multiple readers won't contend with each other. That's fine I guess, good enough. In golang it doesn't seem there's any default immutableMap but you're guarnanteed that concurrent reads of Map won't violate thread safety, and since loadRedirectRules() is done at main before any the proxy is ready to serve, there shouldn't be any issues using a map directly. But probably not worth changing, read-only mutex should be cheap.

 

[Wowfunhappy]

If there are redirects, it seems to reconnect to the redirected hostname. This seems weird, why not check for redirects before the initial outgoing dial in line 1355?

There's multiple lookups to redirectRules[domain] in there, why? You only need to do it once.

Because we don't actually know which host we're connecting to!

Let's say I set up a redirect:

https://foo.com/bar/
https://baz.com

If the client requests the foo.com homepage, the proxy should in fact send back the foo.com homepage, because the request doesn't match my redirect rule for foo.com/bar/. If foo.com doesn't exist, the proxy needs to return a `no such host` error.

However, if the client requests foo.com/bar/, the proxy needs to return baz.com. Notably, the proxy needs to return baz.com even if the foo.com domain doesn't exist or otherwise can't be reached.

At the time of that initial outgoing dial, all the proxy knows is that the client has requested some URL on the host foo.com. The proxy does not know the full URL yet, so it doesn't know whether the client will ultimately need to be redirected to a new host. The proxy might need to send back foo.com or it might need to send back baz.com; it won't know which one until it MITMs the connection, and even then it finds out pretty late in the MITM process.

Making this all work is critical to fixing Help Center, because we don't want to redirect all of help.apple.com to archive.org, we just want to redirect the one resource that's broken. And, although I gave up on trying to fix Apple Maps for now, many of the old Apple Maps subdomains don't exist anymore, so I needed to get past the initial dial in order to redirect them.

The current logic actually still kind of breaks under one scenario: if the original domain doesn't exist and the target domain does, but the specific URL you're trying to access does NOT match the redirect (so the proxy ends up sending you to the original, nonexistent domain), the error you see in your browser is "the server unexpectedly dropped the connection" instead of "no such host". I decided that was okay.

I'm kind of hoping I can find more uses for URL redirects in the future, because I wasted a lot of time here.