Recent Windows 11 clients no longer able to connect to Mac-hosted File server

[Makosuke]

I may be having so much trouble Googling up a solution because this is a niche configuration, so there may be an obvious explainer somewhere that I'm just not digging up, but in any case:

At work we have a file server running macOS 12 (yes, it's ancient, OS upgrade is a separate issue). This has worked swimmingly for a long time with a mix of Mac and Windows clients. However, at some point in the last couple months, I believe after an update got pushed to the Win clients, Windows 11 clients can no longer connect to the Mac server. Mac clients, and Win10 clients are both fine.

The specific error is "Windows can't find [servername]. Check the spelling and try again." However, I can ping the server from the same computer, get the same error when trying to connect via IP address, and it works fine from other clients on the same network, so I'm confident that the actual issue isn't name resolution, it's something lower-level.

These Windows clients are managed by a university, and based on some "similar but not quite the same" situations others have run into, I suspect the issue is that Win11 is now requiring a particular kind of authentication for SMB connections, either by default or due to a group policy that was pushed, and when the server doesn't recognize that encryption or login method, Windows drops the connection and throws a confusing error. Unfortunately, even if I'm right, I don't know what exactly is going wrong or even how to get a more useful log of the failure.

Thus, figuring out what I need to futz with on the client or server side by searching has thus far not come up with anything helpful. Anyone else run into this? Any suggestions?

 

[maflynn]

I'm going to say that

At work we have a file server running macOS 12
...
These Windows clients are managed by a university

The university maybe tightening controls and security.

What version of smb are you using? From what I've read, smb 1 is very insecure, smb 2 lacks encryption and your university may have pushed out updates to enforce smb 3, but since windows 10 is no longer active those updates may not be available (which is why the connections are working).

 

[Makosuke]

Thank you for the reply.

I’ve checked that Mac clients are connecting using SMB 3.0.2, but since Win11 negotiation is dying so early in the process I’m not sure which format it’s trying to connect as.


My suspicion is that whatever change happened (either as a Windows update—we tend to be about a year behind major releases—or new group policy), it’s requiring some features not supported in 3.0.2, so perhaps the solution is as simple as an OS upgrade, assuming newer macOS versions do support newer SMBv3 versions.

The frustrating part is that University IT said nothing was changed, couldn’t figure it out, and punted back to me, despite the evidence that something did, in fact, change.

 

[Makosuke]

Just updating with a few other things I've experimented with:

This Apple article describes something suspiciously similar-sounding, but is specific to a Mac bound to Open Directory, which this one is not.

And this Apple article talks about how to limit the Mac server to particular flavors of SMB.

A Wireshark capture of the Win client trying to connect shows the client sending a Negotiate Protocol Request, and the server replying offering the dialect "SMB2 wildcard (0x02ff)". Then the Win11 client replying with another Negotiate Protocol Request that Wireshark says is malformed, then the server replying with a "STATUS_INVALID_PARAMETER" error (which kinda makes sense if the request is indeed malformed).

Thinking that maybe the client is insisting (in some nonstandard way) on a flavor of SMBv3 when the server said it has SMBv2, I tried two things:

One, setting the -Smb2DialectMax and -Smb2DialectMin parameters on the Windows SMB client to various values (max and min 202, max and min 302) to try and force it to accept a lower version, none of which made any difference.

And two, I tried setting the ProtocolVersionMap on the server to # 4, which is supposed to be v3 only. However, the negotiation capture still shows the same "SMB2 wildcard (0x02ff)" response from the server, so that also didn't seem to help.

I also made sure that the user on the server side matches the user on the client side, and that the "Windows File Sharing" checkbox is checked on the server settings for the user in question, none of which made any difference.

 

[eyoungren]

Just updating with a few other things I've experimented with:

This Apple article describes something suspiciously similar-sounding, but is specific to a Mac bound to Open Directory, which this one is not.

And this Apple article talks about how to limit the Mac server to particular flavors of SMB.

A Wireshark capture of the Win client trying to connect shows the client sending a Negotiate Protocol Request, and the server replying offering the dialect "SMB2 wildcard (0x02ff)". Then the Win11 client replying with another Negotiate Protocol Request that Wireshark says is malformed, then the server replying with a "STATUS_INVALID_PARAMETER" error (which kinda makes sense if the request is indeed malformed).

Thinking that maybe the client is insisting (in some nonstandard way) on a flavor of SMBv3 when the server said it has SMBv2, I tried two things:

One, setting the -Smb2DialectMax and -Smb2DialectMin parameters on the Windows SMB client to various values (max and min 202, max and min 302) to try and force it to accept a lower version, none of which made any difference.

And two, I tried setting the ProtocolVersionMap on the server to # 4, which is supposed to be v3 only. However, the negotiation capture still shows the same "SMB2 wildcard (0x02ff)" response from the server, so that also didn't seem to help.

I also made sure that the user on the server side matches the user on the client side, and that the "Windows File Sharing" checkbox is checked on the server settings for the user in question, none of which made any difference.

I did a Google search based on your Wireshark errors and both of the searches I made suggest turning off SMB signing on the Win 11 computers. It may be that the Win11 machines are requiring a level of signing that MacOS 12 cannot provide, and so it errors out.

What happens if you turn that off (on the W11 machines)?

 

[Wowfunhappy]

I don't think this is going to work, but does it help if you add

signing_required=if_required

to /etc/nsmb.conf on the Mac?

I need this on my Macs but I run a much older version of macOS than you do. (And you thought macOS 12 was ancient, lol...)

 

[Makosuke]

I much appreciate the suggestions, and messing with code signing on both the client and server are things that I had tried without success. It turns out that the actual problem was similar but simpler:

Rather than code signing requirements, IT apparently pushed a Group Policy update without the frontline people being aware. This policy set the minimum version of SMB as 3.1.1 (SMBv3 dialect 3.1.1, that is). Which would be fine, except the macOS SMB server, at least through 26.5.1, only supports SMB 3.0.2. There's a lot of confusing info floating around because the macOS client supports connecting to 3.1.1 servers, but the server doesn't support clients that will only accept 3.1.1.

Because the policy wasn't visible to the user without doing a GPResult scan, and was silently overriding any manual settings I was trying to see if it fixed the issue, I was mistakenly thinking that I had manually set the max SMB version lower than 3.1.1 when it in fact was not. This was severely exacerbated by the completely unhelpful and wrong Windows error message--if it had just said "server does not support minimum client protocol version" or something, it would have been so much easier to figure out what was going on.

It also turns out that the malformed Wireshark packets were caused by me manually setting the maximum SMB version to 3.0.2; apparently when the Group Policy conflicts with that, Windows sends a request that doesn't make sense. Although, the result is the same even when it's configured right, the server just gives a slightly different response then closes the connection and Windows throws the same useless error message.

Because this was a Group Policy on a managed machine, and the Mac server simply doesn't support 3.1.1, there is nothing server-side that could have fixed this (other than waiting for some hypothetical future macOS version that supports 3.1.1), nor anything a user, even with admin privileges, could do on the client side. The solution ended up being to convince the Powers That Be to loosen the restriction and allow connections via 3.0.2.

I hope this helps someone else trying to search for a solution in the future.