Severe Spotlight reindexing problems - root cause

[Ekim Yam]

With the latest beta updates, I noticed significant lags in UI responsiveness generally using my Mac. At times the system would appear to hang forcing a reboot, which would fix the problem for a while but then the laggy performance would return.

I used my AI tool to research forums to determine root cause and this lead to writing a performance monitor. The performance monitor surfaced many root cause issues but long story short surfaced a number of architecture and coding issues with the various tools that make up Spotlight indexing.

My hypothesis is that Apple is upgrading Spotlight functionality to support promised Apple AI functionality and is running into problems caused by the legacy design.

Attached is a more detailed report from an app that I built to determine what’s causing my system slowdowns.

A short term strategy is to limit the scope of Spotlight indexing Apple needs to address the architectural issues.

Since I am unable to attach a file due to the bug, I will paste the report into description. (Hoping that there are no size limitations)

# Spotlight Index Rebuild Performance Issues on Apple Silicon — Feedback Report

## System Configuration

- **Mac:** MacBook Pro (Mac16,6), Apple M4 Max, 16 cores (12P + 4E), 128GB RAM
- **macOS:** 26.5 beta (25F5042g), Spotlight version 2418.5.3.100
- **Storage:** 1TB APFS SSD, ~350GB free
- **Cloud providers:** iCloud Drive, Dropbox, Google Drive, OneDrive (2 accounts)
- **Index scope:** ~420,000 objects, 2.8GB index size

## Issue Summary

A Spotlight index rebuild triggered on April 11, 2026 has been running for **39.6 hours** and is still in the compaction phase with **no sign of completion**. The non-compacted generation count is *increasing* (from 18 to 24), not decreasing. mds_stores has consumed **19.2 hours of CPU time** (49% efficiency vs wall clock) on a machine with 16 cores and 128GB of RAM, indexing only ~420,000 objects on a 1TB SSD.

For reference, this machine can compile a 40-file Swift project in under 30 seconds. A full-disk index of ~420K objects should not take 40+ hours.

## User Impact: System-Wide Application Hangs

During the 40+ hour compaction, **any application that uses NSOpenPanel or NSSavePanel beachballs** when the user attempts to open or save a file. The spinning wait cursor persists for 10-60+ seconds, and the application becomes unresponsive.

**Root cause:** macOS Open/Save dialogs call into Spotlight on the main thread to populate file lists, show recent files, and enable search. With mds_stores at 99% CPU performing compaction and 13.7M pending FSEvents, these queries block waiting for a response. Additionally, if the dialog enumerates a cloud-synced directory, fileproviderd (also at 85-95% CPU) must materialize dataless file metadata, compounding the delay.

**Affected workflows:**
- File > Open in any application (Pages, Preview, Xcode, etc.)
- File > Save As / Export dialogs
- Any drag-and-drop target that invokes a file browser
- Finder itself when navigating cloud-synced directories

**User workarounds (none are acceptable for normal use):**
- Drag and drop files from Finder directly into the app instead of using File > Open
- Use terminal: `open -a "AppName" /path/to/file`
- Press Cmd+Shift+G in the Open/Save dialog to type a path directly (bypasses Spotlight search)
- Avoid cloud-synced folders (~/Library/CloudStorage/) during the rebuild
- Add cloud storage folders to System Settings > Siri & Spotlight > Spotlight Privacy

**This effectively makes the machine unusable for normal productivity work for the duration of the rebuild.** A 40+ hour rebuild means nearly two full workdays where basic file operations hang. On a $5,000+ professional workstation with 16 cores and 128GB RAM, this is an unacceptable degradation.

## Observed Issues

### 1. Single-Threaded Index Compaction

**Problem:** mds_stores performs index generation merging on a single thread, utilizing at most ~100% of one CPU core (occasionally spiking to ~195%, suggesting a second thread for I/O). On a 16-core M4 Max, this means 94% of available compute capacity sits idle during the most CPU-intensive phase of a rebuild.

**Evidence:**
- Sustained 80-99% single-core CPU usage over 40 hours
- Maximum observed: 195% (2 cores) during peak merge activity
- 14 remaining non-compacted generations in the main file index that must be merged sequentially
- System load average of 8-11 (mostly from fileproviderd, not parallelized mds_stores work)

**Impact:** What should be a compute-bound operation on a 16-core machine is artificially constrained to single-core throughput. A parallelized merge strategy (well-documented in LSM-tree literature, e.g., ICPP 2025 "Revisiting Multi-threaded Compaction in LSM-trees") could reduce compaction time by 4-8x on this hardware.

### 2. Non-Converging Generation Count

**Problem:** The non-compacted generation count is increasing over time instead of decreasing toward zero. This suggests the merge process creates new intermediate generations faster than it consolidates existing ones.

**Evidence (index_c — the main file index):**

| Time | Non-compacted Gens | Compacted Gens | Objects |
|------|-------------------|----------------|---------|
| Apr 12 11:14 | 6 | 0 | ~52K |
| Apr 12 14:07 | 8 | 0 | 233K |
| Apr 12 20:05 | 9 | 0 | 349K |
| Apr 13 04:17 | **14** | **0** | 405K |

After 40 hours, index_c has **zero compacted generations** and the non-compacted count has more than doubled. The merge algorithm appears to be making net-negative progress on the main index store.

### 3. FSEvents Feedback Loop with fileproviderd

**Problem:** mds_stores and fileproviderd both consume and produce FSEvents, creating an amplification loop. Spotlight writes index metadata (generating FSEvents), fileproviderd observes those events and performs sync operations (generating more FSEvents), which Spotlight then processes.

**Evidence:**
- fileproviderd runs at 85-95% CPU continuously alongside mds_stores at 80-99%
- Killing fileproviderd: it restarted immediately at 95% CPU (not zombie-driven — real iCloud sync work)
- SIGSTOP on fileproviderd: mds_stores throughput did NOT improve (still single-core bound), but system load dropped 25%
- 13.7 million pending file change events in the FSEvents queue
- `count_add_request = 0` across all stores — mds_stores is not processing new files, just reorganizing, yet the FSEvents queue remains massive

**Impact:** Two system daemons compete for I/O bandwidth with no coordination or priority arbitration. Neither process yields to the other. There is no documented mechanism for mds_stores to signal "I'm rebuilding, defer sync" or for fileproviderd to signal "I'm materializing files, defer indexing."

### 4. Infrequent Heartbeat Updates (~90-Minute Intervals)

**Problem:** During compaction, macOS only updates the CoreSpotlight heartbeat.plist file approximately every 90 minutes. This makes it nearly impossible for users or monitoring tools to determine whether compaction is making progress, stuck, or stalled.

**Evidence:**
- Heartbeat update intervals observed: 96min, 73min, 93min, 72min gaps
- Between updates, all metrics (objects, size, generations) appear frozen
- CPU time continues accumulating, confirming active work, but no progress indicator is available
- Users must resort to comparing `ps -o cputime` output to distinguish "working" from "stuck"

**Impact:** Users experiencing high CPU from mds_stores have no way to determine whether to wait or intervene. Forum posts consistently show users prematurely flushing or rebuilding indexes because they cannot tell if Spotlight is making progress, restarting the entire multi-day process.

### 5. Background QoS Scheduling Limits Throughput

**Problem:** mds_stores runs at background QoS (quality of service level 9), which on Apple Silicon restricts execution primarily to Efficiency cores at reduced clock speeds. For a user-initiated rebuild that makes the system unusable for days, background scheduling is inappropriate.

**Evidence:**
- Howard Oakley (Eclectic Light Company) documents that background QoS tasks run "almost exclusively on E cores"
- On M4 Max, E cores run at ~2.2GHz vs P cores at ~4.5GHz — a 2x throughput penalty
- mds_stores CPU time (19.2 hours) vs wall clock (39.6 hours) = 49% efficiency, consistent with E-core scheduling and I/O contention
- The user explicitly initiated a rebuild via `mdutil -E /` — this is not a background maintenance task, it's a user-requested operation that should receive appropriate priority

**Requested behavior:** When the user explicitly triggers a rebuild, mds_stores should escalate to user-initiated QoS and utilize P cores until the rebuild is complete.

### 6. No User Visibility into Rebuild Progress

**Problem:** macOS provides no built-in UI to show Spotlight rebuild or compaction progress. System Settings > Siri & Spotlight shows no progress indicator during rebuilds. `mdutil -s /` only reports "Indexing enabled" with no progress percentage or estimated time remaining.

**Evidence:**
- We built a custom monitoring app (MikesMacPerfMonitor) that reads the CoreSpotlight heartbeat.plist to extract generation counts, object counts, and index sizes — this data exists but Apple does not surface it
- The heartbeat.plist contains detailed per-store metrics (objects, sizes, generations, merge timestamps) that would be trivial to display in System Settings
- Users currently have no way to distinguish a healthy 40-hour compaction from a stuck/looping process without third-party tools or terminal commands

## Technical Root Cause Analysis

The fundamental issue is an **architectural mismatch between Spotlight's indexing design and modern Apple Silicon hardware**:

1. **The index merge algorithm was designed for single-core machines** and has not been updated to exploit the parallelism available on M-series chips. LSM-tree compaction parallelism is a solved problem in database engineering (RocksDB, LevelDB, Cassandra all support multi-threaded compaction).

2. **The FSEvents subsystem has no coordination primitive** for consumers to negotiate priority. When two high-throughput consumers (Spotlight and fileproviderd) both subscribe to the same event stream and both generate events that the other consumes, there is no backpressure mechanism to prevent cascading amplification.

3. **The QoS system treats all index maintenance as background work** regardless of whether the user explicitly requested it. A user who runs `mdutil -E /` is waiting for the rebuild to finish — this should not be scheduled at the same priority as Time Machine deduplication.

4. **The heartbeat update interval appears hardcoded** at ~90 minutes with no consideration for monitoring or observability. Modern systems typically provide sub-minute telemetry for long-running operations.

## Reproduction Steps

1. On a Mac with Apple M4 Max (or similar), configure 4+ cloud sync providers (iCloud, Dropbox, Google Drive, OneDrive)
2. Run `sudo mdutil -i off / && sudo mdutil -E / && sudo mdutil -i on /`
3. Monitor with: `watch -n 60 'ps -o pcpu=,cputime= -p $(pgrep mds_stores); stat -f "%Sm" ~/Library/Metadata/CoreSpotlight/heartbeat.plist'`
4. Observe mds_stores consuming 80-100% of one core for 24-48+ hours
5. Check heartbeat generation counts — they may increase rather than decrease

## Expected vs Actual

| Metric | Expected (420K objects, 1TB SSD, M4 Max) | Actual |
|--------|------------------------------------------|--------|
| Rebuild time | 1-4 hours | 40+ hours (ongoing) |
| Compaction time | 2-6 hours | 24+ hours (ongoing) |
| CPU utilization | Multi-core during rebuild | Single core (6% of capacity) |
| Generation convergence | Decreasing toward 0 | Increasing (18 to 24) |
| Progress visibility | Progress bar in System Settings | None |
| fileproviderd coordination | Deferred during rebuild | Competing at 85-95% CPU |

## Requested Improvements

1. **Parallelize index compaction** — use multiple threads for generation merging on multi-core systems
2. **Add FSEvents consumer coordination** — allow mds_stores to request I/O priority during rebuilds, deferring fileproviderd sync
3. **Escalate QoS for user-initiated rebuilds** — use P cores when the user explicitly requests a rebuild
4. **Add rebuild progress UI** — surface heartbeat data (objects, generations, estimated completion) in System Settings
5. **Increase heartbeat frequency during active operations** — update every 5-10 minutes, not 90
6. **Implement backpressure between mds_stores and fileproviderd** — prevent the FSEvents feedback loop

## Monitoring Data

Detailed time-series data collected via MikesMacPerfMonitor:

| Timestamp | Objects | Index Size | Gens (C/T) | mds CPU | fpd CPU | mdfind | Phase |
|-----------|---------|------------|------------|---------|---------|--------|-------|
| Apr 11 12:42 | 0 | 0 | 0/0 | 90% | 93% | 0 | Rebuilding |
| Apr 12 06:36 | 268,600 | ~1.5GB | 0/20 | 182% | 86% | 268,600 | Compacting |
| Apr 12 11:14 | 51,902 | 267MB | 0/20 | 81% | 86% | 25,913 | Compacting |
| Apr 12 14:07 | 73,415 | 422MB | 0/21 | 83% | 86% | 25,913 | Compacting |
| Apr 12 15:28 | 118,921 | 717MB | 3/21 | 66% | 86% | 25,914 | Compacting |
| Apr 12 17:38 | 248,121 | 1,334MB | 2/20 | 85% | 93% | 25,913 | Compacting |
| Apr 12 20:05 | 348,778 | — | 2/18 | 85% | 91% | 25,913 | Compacting |
| Apr 13 04:17 | 419,936 | 2,823MB | 2/24 | 99% | 85% | 25,910 | Compacting |

*Note: Object count fluctuations (268K to 52K to 420K) reflect heartbeat.plist tracking CoreSpotlight stores separately from the main Spotlight index. The mdfind count (searchable files) has remained stable at ~25,900.*

## References

- Howard Oakley, Eclectic Light Company: "A deeper dive into Spotlight indexes" (2025-07-30)
- Howard Oakley, Eclectic Light Company: "When and how to rebuild Spotlight indexes" (2024-11-19)
- Michael Tsai: "Runaway Spotlight With Pages on iCloud Drive" (2025-12-02)
- WWDC21: "Sync files to the cloud with FileProvider on macOS"
- ICPP 2025: "Revisiting Multi-threaded Compaction in LSM-trees"
- Apple FSEvents Programming Guide

---

*Report generated 2026-04-13 using data collected by MikesMacPerfMonitor v0.6.0 over a 40-hour monitoring period.*

 

[svenmany]

I'll be interested to read any updates you have when the OS is out of beta. Since you're running the beta version, I assume you're following through and reporting this to Apple.

 

[!!!]

A Spotlight index rebuild triggered on April 11, 2026 has been running for **39.6 hours** and

Just delete the root .Spotlight folder, restart, and let it run from scratch. Don't bother letting it try to fix itself, it will end up taking longer (or never finishing) than just reindexing from scratch.

mds_stores has consumed **19.2 hours of CPU time** (49% efficiency vs wall clock) on a machine with 16 cores and 128GB of RAM, indexing only ~420,000 objects on a 1TB SSD.

For reference, this machine can compile a 40-file Swift project in under 30 seconds. A full-disk index of ~420K objects should not take 40+ hours.

Spotlight only uses E-cores. It shouldn't take very long to index your internal SSD, even with the E-core limit.

During the 40+ hour compaction, **any application that uses NSOpenPanel or NSSavePanel beachballs** when the user attempts to open or save a file. The spinning wait cursor persists for 10-60+ seconds, and the application becomes unresponsive.

Maybe there is an issue with the sharefilelists?

**Root cause:** macOS Open/Save dialogs call into Spotlight on the main thread to populate file lists

I doubt that, did you actually sample the process or is this just AI hallucination?

FSEvents

Delete that root directory as well if it keeps consuming massive amounts of CPU, it's possible for it to get corrupted as well.

Spotlight does not produce (many) fsevents, though.

2. Run `sudo mdutil -i off / && sudo mdutil -E / && sudo mdutil -i on /`

This does not delete the actual root .Spotlight directory, nor do you reboot between disabling an enabling it. If there's something corrupt, it will just persist. You need to delete the directory, restart, and then let it index.

 

[svenmany]

I've looked at some of your assertions and wonder how you deduced them. Do you have access to source code? One example:

**The index merge algorithm was designed for single-core machines** and has not been updated to exploit the parallelism available on M-series chips.

How do you know this? It could just be a bug in something they did update. I see what appears to be lots of parallelism in Spotlight with 4 efficiency cores sharing pretty equally during indexing.

**Problem:** mds_stores and fileproviderd both consume and produce FSEvents, creating an amplification loop. Spotlight writes index metadata (generating FSEvents), fileproviderd observes those events and performs sync operations (generating more FSEvents), which Spotlight then processes.

Of course that would be an astonishing oversight by Apple.

Do you have any Time Machine volumes mounted? I've seen significant issues (not as thoroughly investigated as yours) that seem a bit similar when Spotlight indexes Time Machine volumes.

 

[Ekim Yam]

I'll be interested to read any updates you have when the OS is out of beta. Since you're running the beta version, I assume you're following through and reporting this to Apple.

Yes, i reported it to apple, but no response yet. They don't seem to respond to individual requests though based on my past experience.

 

[Ekim Yam]

Thank you for your feedback.
I have a high-capacity machine (M4ProMax 128G) and this is taking far too long. I expect Apple to diagnose this and provide a solution. Apple provides almost no feedback to help users understand what's happening — no progress bar, no generation counts, no indication of what's causing the spotlight problems. The heartbeat.plist data exists but isn't surfaced anywhere. The information available on forums — including my own post — is mostly inaccurate, which helps no one trying to solve this. But feedback from knowledgeable contributors like yours helps. I'll update this thread if the recommendations resolve the issue.

The original compaction ran for 46 hours on an M4 Max (128GB, 1TB SSD) indexing ~420K objects

- The non-compacted generation count increased from 6 to 14 over that period — it was going backwards
- Zero compacted generations in index_c after 46 hours
- A second rebuild with Spotlight exclusions (cloud storage, caches, homebrew excluded) is now 2 hours in and already further along than the first attempt was at hour 20. I will monitor that and apply your additional recommendation if this one stalls.


I don't have access to Spotlight source code. I used an AI tool to help monitor and diagnose the situation, and we made inferences that were stated in my note as facts. Specifically: - The "FSEvents feedback loop" claim was an overstatement. You're right that Spotlight is primarily an FSEvents consumer and a bidirectional amplification loop would be an obvious design flaw Apple would have caught. - The "single-threaded by design" claim was an inference from observed CPU usage, not from source code analysis. Your observation that indexing does distribute across E-cores is noted — compaction's limited parallelism could be a bug, not a design choice.
- The NSOpenPanel beachball analysis was a timing correlation, not a confirmed stack trace. The sharefilelists suggestion is a good lead I hadn't considered.
- You're right that mdutil -E doesn't delete .Spotlight-V100/. I'll run the full delete + reboot approach as an option.

The AI tool helped get me to a certain point but this exercise shows the AI limitations that are exasperated by poor quality information - from Apple and the millions of general posts that the AI tools draw upon.

Again, thank you for your valuable feedback that helps me get my machine back to normal performance.

 

[svenmany]

It's very hard to learn much given how little Apple shares.

I'm eager to hear about anything else you discover. Certainly during my struggles with Spotlight indexing Time Machine volumes, there did seem to be a large number of index generations accumulating, which was quite a mystery for a brand new volume. The first time it was indexed, there was just the single generation. On reattaching the volume, Spotlight would reindex and the generations would blow up to well over 20. Some kind of feedback bug could have explained that, but it just never occurred to me that file providers would even care about FSEvents related to a Spotlight index changes.

Best of luck.