Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Other A proxy to fix HTTPS issues on old versions of OS X

Quick note: yesterday's update actually broke the Chromium Legacy Downloader PrefPane. I imagine a lot of people are using both. Please redownload the PrefPane.

Sorry, this gets a little complicated because the PrefPane needs to work under a bunch of different scenarios, and it needs to actually use https properly, since I don't want anyone downloading a malicious version of Chromium...
 
Wowfunhappy said:
Yep! Although, if it's already been installed than it shouldn't actually do anything regardless of whether you check the box. (I'm pretty darn sure I didn't mess anything up this time! 🤞)

------

@Macbookprodude I did not make this for LWK, I made it for other apps like Dashboard, Dictionary, QuickTime, etc. Actually, every time someone tells me they’re using this proxy in order to run an outdated web browser (including LWK, last updated in 2018), I feel a little uncomfortable, because I’m concerned about security. (But, it’s your computer and not mine, so make your own choices; I don't particularly want to get into a discussion about this.)

As for OS9—yes it can, but please figure it out yourself. I've already done my absolute best to explain it to you in the PPC section, and I cannot do it any more.
Click to expand...
Well for OS 9, I use Browservice which my 2012 MacBook Pro is hosting. As for dashboard, when did Apple pull the plug on that ? I remember all my widgets worked recently and now they aren’t except for my printer driver widget.
 
Macbookprodude said:
As for dashboard, when did Apple pull the plug on that ? I remember all my widgets worked recently and now they aren’t except for my printer driver widget.
Click to expand...

The built-in widgets broke over the course of 2019 and 2020. I fixed two of them, and I took steps to ensure they can function without a proxy: https://jonathanalland.com/dashboard-widgets.html

For third party widgets, it of course depends on whether the third party servers are still up. But in some cases, the servers are up but require a proxy. For instance, Delivery Status and Notefile: https://junecloud.com/software/dashboard/
 
Wowfunhappy said:
On OS X 10.9 and below, the package will automatically set the environment variable HTTPS_PROXY to localhost:3128 in /etc/launchd.conf.
Click to expand...
Okay, now I'm having misgivings about the fact that I did this.

It fixes curl and git, which is great. But if you have any command line tools which respect this environment variable but not Keychain Access—including a lot of UNIX software built by MacPorts—they'll break because they don't see the Squid certificate, unless you manually unset the HTTPS_PROXY environment variable and/or the path to the certificate.

Furthermore, it means that if you turn off the proxy in System Preferences, some software will still use it. I'm basically concerned that I'm breaking an implicit promise to the user, about when the proxy will be used.

On the other hand, it does fix software. I don't know another way to make Git Tower work, for instance.

Does anyone have strong opinions on this?
 
I don't know if that counts as "a strong opinion" and a CLI tool but I use youtube-dl. I do know that many apps (e.g., Dropzone by Apptonic) use CLI tools written in Ruby: from that, I can infer the similar use of other CLI tools written in other languages such as Perl or Python. Will this change affect them?
 
Let me elaborate on this. In order for software to connect to a modern HTTPS server, one of the following things needs to happen.

  1. The software natively supports modern HTTPS, and connects without using our proxy.
  2. The software connects via our proxy, and trusts the Squid certificate.

Simple, right? Now, let's think of some scenarios that could cause things to go wrong.

  1. The software does not natively support modern HTTPS, and does not use our proxy. Obviously, this isn't going to work. The Dictionary app falls into this category; it works now because we added our own code to make the app do what we want.
  2. The software uses our proxy, but does not see the Squid certificate, and so assumes you're being hacked and does not work.
There are several possible causes for #2. The developer might have intentionally decided to ignore user certificates (certificate pinning), but it could also be because the software relies on its own certificate store rather than Keychain Access.

Software built against non-Apple versions of OpenSSL generally falls into the latter category. Ironically, this software probably would work fine if it didn't use the proxy at all, because modern versions of OpenSSL natively support modern HTTPS protocols.

So, what determines whether an app connects via our proxy? Well, some use the proxy settings in System Preferences, and others look for an HTTPS_PROXY environment variable. The latest update to my proxy package automatically sets the HTTPS_PROXY environment variable so that more apps will use the proxy. However, if some apps read the environment variable, but do not use Keychain Access, they won't find the Squid certificate and will break.

Now, let's look at curl. Curl ignores System Preferences; to make it use a proxy, you need to set the HTTPS_PROXY environment variable.

The built-in Apple version of curl does not support modern HTTPS, but it does use Keychain Access. Without the environment variable set, it won't use the proxy, and won't work. If you set the environment variable, it works perfectly!

Versions of curl that are built with MacPorts are linked against modern copies of the OpenSSL library. This means they natively support modern HTTPS, but ignore Keychain Access. If you don't set the environment variable, it works perfectly. If you set the variable, it breaks.

So, does this effect Ruby, Perl, Python, etc? I'm not sure, but it probably depends on whether you're using the copy that shipped with OS X, or a version linked against its own version of OpenSSL.
 
I can say that python (youtube-dl specifically) breaks when using squid. You have to add the flag --proxy " " for it to work.

Cheers
 
Oh, thanks @Wowfunhappy , that was helpful. However, the only solution that could potentially address this problem adds another layer of complexity: it means either the user or you will have to take this fact into account and add conditionals checking the deployment target for its version: I now skimmed through the curl's man page and found that, quite naturally, it offers an array of proxy options. So, if you leave HTTPS_PROXY as is, then we'll have to deal with it on our own (i.e., pseudocode if [ the_macos_version -le 10.10 ] ; then curl --proxy localhost:3128 ; fi or if [ the_macos_version -le 10.10 ] ; then export HTTPS_PROXY=localhost:3128 ; fi). However, provided only advanced users utilize CLI we can safely assume they'll know how to figure that out. I use youtube-dl with the proxy option. The only suggestion would be to document that aspect of running Squid.
 
Last edited:
wicknix said:
I can say that python (youtube-dl specifically) breaks when using squid. You have to add the flag --proxy " " for it to work.
Click to expand...
...not for me. Are you using the official binaries from their website or something else? I bet if you install it through MacPorts it gets linked differently.

maverick28 said:
However, provided only advanced users utilize CLI we can safely assume they'll know how to figure that out.
Click to expand...
Where this gets more complicated, however, is for GUI apps that use commands like curl under the hood. I'd like those to just work (to the greatest extent possible), because they might be used by people who have no knowledge of the Terminal. An example of an app like this is Git Tower.

My question really comes down to, is it evil to set this environment variable for the user automatically? I think I have a good reason for doing it, but I'm concerned that there's the potential for breaking things without giving the user the knowledge to solve it.
 
Last edited:
Official binaries of youtube-dl. Macports doesn't update it regularly enough. For twitch streams to keep working you need the latest. Older versions may keep working for YouTube, but break for twitch regularly.
 
wicknix said:
Official binaries of youtube-dl. Macports doesn't update it regularly enough. For twitch streams to keep working you need the latest. Older versions may keep working for YouTube, but break for twitch regularly.
Click to expand...
Huh, then I have no clue why it doesn't work out of the box for you. I use youtube-dl to watch almost everything, and I also use the official binaries.
 
Last edited:
Wowfunhappy said:
My question really comes down to, is it evil to set this environment variable for the user automatically?
Click to expand...
No, it's not. That's how deploying software goes. Apple and MS constantly change things and something breaks from time to time. Just today - one more iCloud Mail outage.
BTW, speaking of installing Squid you could automate at least 2 more things: the certificate installation and system restart to effect the change: considering the former Adobe, MS and Apple themselves do that matter-of-factly, nobody complains; considering the latter, at least on 10.7 I had to restart to make Squid run. I think you better include that in the install and post-install scripts.
Also, I had to include 2 more domains in the exclusions list in squid.conf (10.7): myapps.itunes.apple.com and xp.apple.com. Without them, MAS was showing "We were unable to show this item in your store. Try again later." every time I tried to view the information about an item. I perused outgoing connections with Little Snitch to make sure that was the case: once I added these entries the message went and it was able to load the item's room. Perhaps, that doesn't concern macOS 10.9 and later as those use different host routes to the same services.
 
Last edited:
(I have no knowledge in certificates and security)...
While fiddling as root user, I recognized that all certificates got up to date.
Then I logged back to normal and did a new user.
In the new user account, all certificates are up to date, too.
Nevertheless, it has no impact on Safari while opening websites, so it does not help us with Safari.
But what is then primary the problem, because security test sites confirm modern TSL 1.2 ?
 
With Safari it's only a problem of HTML and CSS compatibility even if the connection is a success: the site will have an incorrect layout or go completely blank. As for the version of TLS, for every old Safari you open it in, howsmyssl.com reports TLS 1.3, not 1.2.
Speaking of certificate pinning and the use of Keychain Access this is indeed a legit concern because I experienced that myself without realizing the workings as per @Wowfunhappy's explanation above: with the proxy up and running 10.7 lost location capabilities with the dreaded "Was unable to determine your current location" message in System Preferences. I knew there was a system location demon called locationd however it continuously failed: I verified that by checking the appropriate box under "Time Zone" in the Date&Time pane of Sysprefs with Little Snitch'es Network Monitor launched. Mac OS X 10.7 Lion calls iphone-services.apple.com to obtain a location. But this demon apparently launches only once just to contact that server, and to contact the server it requires an unfiltered connection. It suddenly occurred to me that the problem had smth to do with Squid. Once I turned it off and checked "Use automatic location" locationd came into action and established the required connection so I finally was able to see the red pin placed on the map.

The most interesting thing happened afterwards: I edited squid.conf by including iphone-services.apple.com in the list of excluded domains and restarted. Now, every time I tick/untick "Use automatic location" the connection with iphone-services.apple.com is established but a strange thing is that Little Snitch shows this domain among the domains coming through Squid despite me excluding it from proxying.

Apparently, locationd which is a Unix executable ignores either Squid or - most probably - Keychain Access. Interestingly though, another snag cropped up which is that putting this domain on the list of exemptions under Proxy of Network Settings has no effect whatsoever. It used to but now it doesn't and I fail to trace the exact reason or moment when that changed.

Consider also that macOS is a vast collection of Unix executables under the hood.

P.S. System Preferences-Date&Time-Time Zone now reports Closest City: GEOCityProxyName - (null) while showing a correct location with the red pin on and my city in a flashing dark box (a box with a city name which when "Automatic location" is enabled typically pops up briefly once the pin is placed and then goes away).
 
Last edited:
  • Like
Reactions: Xde
maverick28 said:
The most interesting thing happened afterwards: I edited squid.conf by including iphone-services.apple.com in the list of excluded domains and restarted.
Click to expand...
Note: This will tell Squid to pass on the https packets verbatim, instead of decrypting and re-encrypting them first. They still go through Squid. I wasn't able to totally understand your post, but I think this may be part of the confusion.

maverick28 said:
Consider also that macOS is a vast collection of Unix executables under the hood, so that poses a serious difficulty: how do we take into account all of them?
Click to expand...
Note: it's not as simple as Unix vs non-Unix. OS X is a Unix operating system, so almost everything is a "Unix executable", regardless of whether it is designed to be used via the Terminal. (And if locationd is indeed ignoring the Squid certificate in Keychain Access, it's likely due to intentional certificate pinning. I'd be shocked if an Apple binary had its own certificate store, that would be nuts.)

Anyway, I can exclude iphone-services.apple.com in the default config if that's needed on Lion. Very odd that it seems to be unnecessary on Mavericks.

----

Also, I think I'm going to make the environment variable an optional checkbox in the installer.
 
Wowfunhappy said:
Note: This will tell Squid to pass on the https packets verbatim, instead of decrypting and re-encrypting them first. They still go through Squid. I wasn't able to totally understand your post, but I think this may be part of the confusion.
Click to expand...

I think you understood my post well: I wasn't aware that Squid still operates on all the OS connections even if being told to exclude domains. I didn't know "to exclude" is synonymous with "exempt from encrypting HTTPS packets".

Wowfunhappy said:
Note: it's not as simple as Unix vs non-Unix. OS X is a Unix operating system, so almost everything is a "Unix executable", regardless of whether it is designed to be used via the Terminal.
Click to expand...

Which is what I implied. I forgot to mention that pinging iphone-services.apple.com results in the 100% packet loss regardless of whether the proxy is enabled or not. As well as certificate pinning it could be a consequence of the domain settings: even macOS has the option to turn on absorption of incoming pings which makes your computer ignore any pinging attempts. If it's the case then it could explain these ping failures and certificates have nothing to do with that; it's hard to state with absolute certainty without extensive knowledge provided Apple produced the relevant documentation (and did Apple make every binary use its own certificate store really?). As for locationd, I believe its behaviour is different in different revisions of macOS. In Lion it can start its job not before quering iphone-services.apple.com and fails gracefully if doesn't succeed. The question is whether Squid won't break the connectivity of many binary programs each of which is an unknown entity. If those programs allow proxification via CLI options then those side effects can be circumvented. We had to exclude Apple's software update and distribution domains and I believe it's what Apple recommends for Mac admins configuring network firewalls and VPN (the category which Squid falls under too, to some extent); now, it's some random domains, running a proxy on which can break the process.

Wowfunhappy said:
Anyway, I can exclude iphone-services.apple.com in the default config if that's needed on Lion. Very odd that it seems to be unnecessary on Mavericks.
Click to expand...

It's because Mavericks and everything after have different routing schemes. In Mavericks and Mojave, I see that when I'm toggling Automatic location detection under Time Zone locationd fires up immediately and Little Snitch displays it as a separate connection, not the one handled by Squid like in Lion.
 
Last edited:
Ok, I just did the howsmyssl.com report under LWK with my G4 Titanium - it states good stuff, except for some session support ticket saying improvable - I am not sure what this means. Next will try it in classilla.
 
Okay, sounds like I'll need to add iphone-services.apple.com to the SSL bump exclusion list by default in the next version.

maverick28 said:
I didn't know "to exclude" is synonymous with "exempt from encrypting HTTPS packets".
Click to expand...
It's being excluded from ssl_bump, which is what Squid calls this decryption-re-encryption process.

It makes sense if you think about it. Squid is a proxy server. Other programs send data to Squid, and Squid can decide what to do with that data: modify it, send it to a different server, sent it unmodified, etc. But it can't retroactively stop data from being given to it in the first place.
 
Last edited:
Quick note that today, I found a program called Proxifier which say sit can send specific apps through the proxy, without applying it for the entire system. If it works, it would get rid of the remaining minor weirdness of using the proxy—namely that it can occasionally cause problems in apps that use their own certificate store instead of Keychain Access. Instead of adding the proxy to System Preferences, you'd add it only to problem apps that need it.

Proxifier Legacy Downloads

www.proxifier.com www.proxifier.com

Be aware that:
  • As of this writing, I haven't tried it (need a free day)
  • It costs $40 after a free trial
 
I took a glance. From the screenshots it's too complex to tune up every app and requires extensive knowledge of the networking ins and outs. Another big obstacle is that it's costly – 40 bucks to pay for a utility which serves one purpose and the price almost reaches that of such apps as Pixelmator and Affinity. The Legacy Mac proxy copes with this task really well for free.
 
  • Like
Reactions: RobJos
I use Chromium Legacy for web browsing.

I really don’t know how many times I can say this—I did not make this proxy for web browsing and I do not recommend using it for web browsing, because—while you are free to do whatever you want—I think using the combination of an outdated web browser on an outdated OS is extremely unsafe. Unless you also turn off Javascript, and no one wants to do that.

The purpose of this proxy, in my view, is to fix apps. The Dictionary app, QuickTime, Sparkle, Apple Mail, CodeRunner—even MacPorts if the official mirrors are down and you need to download from https sources.

However, please do also feel free to not use it too! It makes no difference to me.
 
Last edited:
  • Like
Reactions: RobJos
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back