The real reason is that Apple lacks engineering talent. The ones who have talent have job security for life, thanks to Tim’s policy of never laying people off, so they only need to do the bare minimum.
Did you read why it’s faster when connected to the internet? Specifically if there are any online only files (or emails) they need to be downloaded first to be fully indexed
Maybe they could have forced the indexing to start with the applications folder. That doesn’t seem like a lot of work to index, and as I (and more people) ONLY use Spotlight as an app launcher, that would save us some pain.