They should start worrying. iOS 7 is a heaping pile of crap. The phones are yesterday's news with small screens. They can't even make the transition to Haswell in a timely matter. The stores are losing momentum and people mostly go there for repairs and returns. Why should we kid ourselves? Things are not well in Apple land. It's time for Cook to leave...
I don't think you quite understand the market right now.
First, iOS 7 seems to be pretty good. My only complaint is that they let Johnny Ives lock them into too much of a "cartoon" theme that doesn't play nice with user changes (backgrounds, etc) the core features are all pretty good, but no way to "opt out" of all the candy coating. Apple could easily fix that with a theme in the older style in a point update.
Haswell is more dependent on Intel to deliver. Reviews have that the NEXT chip is much better as Haswell is a bit of a "one trick pony" for power consumption and not much more. Apple doesn't need to redesign their line for every lame part Intel makes.
I DO AGREE with stores and traffic. Apple's lack of PRODUCT for two whole quarters means I have DRIVEN PAST my Apple Store that's 30 miles away because there's nothing to see. There's not even "little stuff" like accessories to buy, to justify stopping, it's pretty bleak.
Overall it feels like Apple is Coasting through the whole year. That's not terrible as the current crop of A5-A6 products don't NEED people to run out and buy new... It's been a good year for devs to sell lots of apps to new users. But it's been REALLY SLOW... Especially for Rumor sites there's literally no products to SPECULATE about... Ouch