Spotlight mdworker.shared file flooding system log; removal stops search index from working properly?

[armoured]

What I'd really like to understand is what spotlight is accessing that is driving it quite so crazy. Nothing immediately comes to mind.

Interesting comparison bit: I have three macs, two of them have had this 500k+ empty folders with the constant killsigs. The other one also has the messages but the number of empty folders - a bit less than 4000.

Offhand I cannot think of any significant difference in their systems, applications, and other stuff - even the usual suspects (I think of Adobe...) are almost all the same, although I haven't done formal comparison. The one with the least folders is less actively used, but it's always on for some server stuff and does get used regularly.

I wonder if there's some daemon that's supposed to clean these up periodically that isn't running or is misconfigured. I've also thought about trying a program like Onyx that cleans up caches, might try that later to see.

 

[Brian33]

Yeah, I have three Macs, also, and had high numbers of empty folders on two of them but not on the third. I can't figure out any reason why that would be, either. My (probably) most-used system and my always-on server had the most orphans, though.

I spent some time trying to figure out what files the terminated mdworker processes were accessing, on the hypothesis that some rather small set of particular files was causing the problem. I hoped to get lucky with the 'lsof' (list open files) command, but, nope. Maybe 'dtrace' could help, but I don't know how to use it and I decided it's not worth the trouble to try to learn it, if I even could manage it!

 

[auxbuss]

Just a data-point on Catalina:

I found I had 850k empty directories in: /private/var/folders/*/*/C/com.apple.metadata.mdworker

Since this is a user cache, I simply deleted the directory. That worked fine; the system recreated the folder and then began adding new empty folders to it at the rate of about 3 a minute.

(I used ncdu to ensure there was no data in any of the directories. There wasn’t, which raises a few questions and an eyebrow or two.)

re: /private/var/folders/*/*/T/com.apple.metadata.mdworker

That has 4,000 empty directories. I presume from the number that something is clearing it out, so I've left it alone and will monitor.

edit: I happened to boot into recovery (cmd+r) and discovered that this cleans up the T directory, but not the C directory.

 

[phrehdd]

I had that problem on my MBP 2015 that I recently retired. I'll see what's what with the Mini M1 over time beyond the "AP watchdog has expired" reboot I got (once).

 

[auxbuss]

24 hours after clearing out the C and T directories, the empty directory counts are 2500 and 2200. Both are completely empty; not one byte of data anywhere.

There are a couple of real impacts I've noticed:

Obviously, if I search /private/var/folders, it completes much faster now. For example, an ncdu of that directory had ballooned to a couple of minutes or so; it's now almost instant (one second, perhaps). However, the corollary is that any global file process will be equally longer.

Also, when logging in, rebooting, etc. spotlight does some sort of "indexing". As above, this process had ballooned to a couple of minutes or so. It is now absent. (You can see spotlight indexing [or I could] by opening spotlight after logging in cmd+<space>)

I do wonder whether this bug is the cause of some folk having random "spinning wheels" or other slowdowns. Clearly it needs to be fixed. Has anyone reported it to Apple?

 

[armoured]

Small additional data point: on one computer, both C and T directories had ballooned in size to over 70k directories there. I rebooted to see if any impact - the T directory was now close to nil, the C incrementing still slowly.

Probably wouldn't mind if I didn't have the strong impression that these sigkill mdshared things were causing overall delays, especially if left alone for long periods.

Should probably just figure out how to write a cron to clear these out once a day or something.

 

[incumbent]

Have any of you noticed performance changes in waiting for Open/Save dialogs? Mine have gotten absurdly delayed lately and I have no idea what the root cause would be but i've had some spotlight indexing shenanigans off and on for years

 

[mikedvzo]

Why not just exclude all of /var from Spotlight? I see no need for it.

 

[incumbent]

Why not just exclude all of /var from Spotlight? I see no need for it.

ha i didn't think about that, i'll see if that changes anything for my open/save pinwheeling if nothing else. i use osquery on my workstations and there's a lot of log churn in there including some of my docker volumes that don't need to be indexed either. thanks for the inspiration!

 

[godzfire]

OP here. I just remembered I posted about this awhile back.

I BELIEVE I HAVE FOUND A FIX (or at least a workaround or the cause)

After doing some further digging and comparing of what this thing was spazzing about, and with a little random luck, I found out that it seems to be the private folder that's the culprit. As a test, I added the folder to the Spotlight Privacy tab in settings (aka the Prevent Spotlight from searching these locations list). The messages either immediately stopped or after the first reboot. I've since had no issues.

Sharing this for others to see if they can replicate the success, and that maybe I just found a fix to something that Apple can't (or won't) fix.


EDIT: I wanted to make an edit in my original post to add this at the top, but there's no edit button. Can a mod or staff member help me out please?

 

[Brian33]

As a test, I added the folder to the Spotlight Privacy tab in settings (aka the Prevent Spotlight from searching these locations list). The messages either immediately stopped or after the first reboot. I've since had no issues.

Thanks for posting your experience! Sadly, it didn't work for my system (Mojave). I added '/private' to Spotlight's exclusion list and rebooted. I'm still getting the 'Service exited...' messages and an increasing number of empty directories created. As far as I can tell it didn't make any difference.

Others may have better luck!

 

[ghostlines]

Sharing this for others to see if they can replicate the success, and that maybe I just found a fix to something that Apple can't (or won't) fix.

Tested this method, however the SIGKILL log entries are still constantly showing up.

 

[Tiberinoxum]

I cannot resolve the issue even with a sparkling fresh clean install of Big Sur on my late 2013 MacBook Pro or my 2018 Mac Mini. I have been successful filtering out the entries so they do not show up with the instructions in post #46 above and did not have to disable SIP (but did need sudo) and then restarted.

Tiber

 

[sporkme]

Does anyone else see any responsiveness issues when this happens? I get some freezes - mouse still moves, but some windows go unresponsive and then beachball, things like space switching work.

Whenever this happens and I look in the log, I've got a bunch of the "com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0B000000-0000-0000-0000-000000000000[17782]): Service exited due to SIGKILL | sent by mds[74]" messages.

 

[Edgecrusherr]

I was having this issue on my M1 Mac mini. In my case, I have some large server shares mounted. I guess Spotlight hasn't been able to index them in the last 4-5 month that I've had this computer. I added the shares to the Spotlight exclusion (privacy) list in System Preferences, and it stopped right away. More of a band-aid, than a fix. I'll try turning it back on after I run the 11.5.1 update.

 

[Tiberinoxum]

Does anyone else see any responsiveness issues when this happens? I get some freezes - mouse still moves, but some windows go unresponsive and then beachball, things like space switching work.

Whenever this happens and I look in the log, I've got a bunch of the "com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0B000000-0000-0000-0000-000000000000[17782]): Service exited due to SIGKILL | sent by mds[74]" messages.

I have no noticeable issues with performance when the messages are piling up in the log.

Tiber

 

[iamaydo]

Does anyone else see any responsiveness issues when this happens? I get some freezes - mouse still moves, but some windows go unresponsive and then beachball, things like space switching work.

Whenever this happens and I look in the log, I've got a bunch of the "com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0B000000-0000-0000-0000-000000000000[17782]): Service exited due to SIGKILL | sent by mds[74]" messages.

I have no noticeable issues with performance when the messages are piling up in the log.

Tiber

Same here, on a M1 MacBook Pro. However, I do wonder if it has anything to do with Messages app on Mac being forever stuck indexing.

 

[cwolf127]

I ran this command to remove all those empty directories:

find /private/var/folders/*/*/C/com.apple.metadata.mdworker -type d -exec rm -rf {} \;

...but still getting continuous

Service exited due to SIGKILL sent by mds

 

[auxbuss]

I ran this command to remove all those empty directories:

find /private/var/folders/*/*/C/com.apple.metadata.mdworker -type d -exec rm -rf {} \;

Here's what I use:

rm -rf ${TMPDIR/T\//}'C/com.apple.metadata.mdworker'

 

[auxbuss]

Similarly, this to list the size of the empty C and T directories:

ls -l ${TMPDIR/T\//}'C/com.apple.metadata.mdworker' | wc -l &&
ls -l $TMPDIR'com.   apple.metadata.mdworker' | wc -l

 

[builtinbc]

I have the same issue with the log bombing with errors of the form:
Sep 8 22:53:29 MyMacBookPro com.apple.xpc.launchd[1] (com.apple.mdworker.shared.0D000000-0600-0000-0000-000000000000[5697]): Service exited due to SIGKILL | sent by mds[3756]

However, I do not have any directories named "com.apple.metadata.mdworker", as per searching in Spotlight with System folders and invisible files and folders included, and in Terminal:

ls: /private/var/folders/*/*/*/com.apple.metadata.mdworker: No such file or directory

Running the latest regular release of Big Sur (11.5.2). Has Apple renamed this directory in a recent release?

 

[Brian33]

Edit: Oops — I just realize that you used the extra * in place of the C. Seems like that should work. Maybe you are right about Big Sur.


You’ve got one too many asterisks in your command line; try
ls -l /private/var/folders/*/*/C/com.apple.metadata.mdworker | wc -l

Although it is certainly plausible that Big Sur changed the location. I can’t check that at the moment. I don’t think that spotlight will work — I read somewhere that it only indexes files in home directories, and not in system directories.

 

[OffWorld]

I can't believe this bug is still present in 11.5.2, an update that vaguely promises "bug fixes." Not this one apparently. And for me each update has made it worse.

I hadn't fully disabled Spotlight but I added all my drives to the Privacy list and unchecked all the categories. Under 11.5 whenever I plugged in my Time Machine drive it would start flooding the log again until I unmounted the two HFS+ partitions on that same physical drive, even though they were on the Privacy list. Didn't matter how many times I added and removed and re-added them to the list. In 11.5.1 even unmounting them wouldn't stop the mds SIGKILL flood, I'd have to check and uncheck Spotlight categories, then it would stop. Under 11.5.2 even that trick no longer worked, there appeared to be no way to stop it. I began wondering if it was something about the drive having one APFS (Time Machine) partition and two HFS+ partitions. So I reformatted the HFS+ partitions to APFS and Spotlight stopped flooding the log file. I then decided to just delete those partitions and expand Time Machine's to use the drive's full capacity. Again, Spotlight behaved itself.

I have another external drive with two HFS+ partitions that are both on the Spotlight Privacy list and when I plug that in Spotlight remembers not to try indexing them. So it's not an issue with the Privacy list or external drives in general.

Is there a rule I missed somewhere that your Time Machine drive, or drives in general, can't have other partitions in different file formats? Is Spotlight treating it like a corrupted drive or something? Quirk of the M1 Macs?

 

[ImConfusedAgain]

i am also getting this mdworker shared in system log running catalina 10.15.7

 

[OffWorld]

Ok, I'm at my wit's end with Spotlight. As I said, this bug is getting worse with every update! What I posted before only lasted until I rebooted, then it started trying to reindex my Time Machine drive the next time I plugged it in.

It also seems in Big Sur 11.5.2 you CANNOT turn Spotlight off! I went into Recovery, turned off SIP, rebooted, ran the commands to supposedly disable Spotlight, the terminal even says its disabled, but it's NOT. And since either Mojave or Catalina you can no longer exclude your Time Machine drive from Spotlight because it uses mds in some capacity, which is probably why you can't turn it off if Time Machine is enabled. But if I can't turn it off and I can't exclude my Time Machine drive it keeps flooding my log file with "Service exited due to SIGKILL" until I unmount my Time Machine drive.

In other words Spotlight is a useless buggy POS software that now means I *also* can't use Time Machine because it (for some reason) *requires* Spotlight now. I'm going to have to abandon using BOTH built-in Apple programs in the hope that using third-party solutions for backup and search will prevent Spotlight from trying to index things and flooding my log file.

Apple really needs to stop releasing new major versions every year and concentrate on fixing all the bugs in the current release. Especially because this bug has persisted over multiple major releases for YEARS already. I can't believe nobody at Apple is aware of this problem, and if so why can't they seem to fix it?