Well, credit cards can be used in major stores in Japan, but part of it is to do with the prevalence of IC (stored value) cards like Suica, PASMO, ICOCA, etc which have been providing widespread contactless payments for a lot longer than credit cards have in other countries. Contactless IC cards have been around for about 17 years, and their use for transit means a huge proportion of people in Japan have been using them for a long time. They have no barrier to entry (because they aren't a credit card), they're faster to use than credit cards, they're widely accepted for purchases in stores, vending machines, etc, and can be recharged via a credit card or bank account automatically anyway.
So the long term prevalence of IC cards means credit cards have been the ones trying to play catch up. Why would someone use a credit card in Japan when IC cards have been around longer and accepted in way more places? (the "credit" nature of credit cards was also probably a hard sell culturally by banks in Japan after the distrust created by the Japanese financial crisis of the 1990s)
And also years before the iPhone even existed, IC cards were already available on phones in Japan in the form of what's known as Osaifu-Keitai (i.e. mobile wallet) support. Japan was well ahead of other countries in mobile phone technology in the 1990s and early 2000s.
So basically, IC cards were there first and still provide superior convenience today. (edit: I should clarify that all of the above is talking about every day purchases like food, taxis, convenience stores, supermarkets, buses, trains, vendors, and so on - for expensive purchases like buying a TV or other electronics, paying for a hotel, etc, credit cards will almost always be accepted in those situations anyway)
If you visit Japan, do yourself a favour and pick up a Suica card, or if you have an iPhone 8 or Apple Watch Series 3 or above, you can just create one directly in Apple Pay using the Suica app. This is what I did a couple of weeks ago when I was lucky enough to be able to make another visit.
Suica has been available on Apple Pay for a few years (starting with the Japan-specific models of the iPhone 7 and Apple Watch Series 2), and it's proven incredibly popular there. Apple Pay also natively supports the recharge of Suica cards from any Apple Pay Visa, Mastercard or Amex card as well, making it particularly handy for inbound travellers.
