Ever since covid and the parts shortages, along with constrained shipping it appears Apple thinks about larger packages as a negative. So the large boxes that the 27” iMac used were not as valuable as many smaller boxes that other more profitable devices that Apple needed to store/ship. That’s more their thinking then large all-in-one have fallen from grace. So the more valuable a product and the less warehouse space is involved seems their current priorities.
That really doesn't track well. The Mac Studio box + Studio Display box cost more to ship than just an iMac box. It isn't warehouse space. Apple is probably trying to cut down on how much jet fuel they burn flying Macs around the globe from single production factory, but not the real driver here.
The PC getting smaller (and faster) is not new. If has been happening since the 70's. As long as the fab process improvements do not completely stall, it will continue to happen across the PC industry.
Apple used to sell laser printers. As the competition got much higher while quality improved , they dropped out. Similar trend is happening in the large monitor space. Apple is retreating into higher mark up 'display docking monitors' more and more. They probably won't completely quit like in the laser printer space as need a monitor to complete a GUI computer. ( hard to have GUI if can't see anything

). But it is clear Apple is not trying to sell large monitors to every Mac user .
iPhone , iPad , laptop the display panel is inherently part of the device. ( also Watch). Apple will drive those 'wins' for panel unit sales and attachment. So 13-14" iPad probably before an > 27" iMac.
As for profitable ... the Mac Pro and XDR are extremely likely profitable for Apple. In fact, the margins are so large that they don't need to see high volumes to make substantive money. They all have a 'low volume' tax on them to make their margins appealing to Apple. If the large screen iMac has relatively low volume ... it probably gets the tax also. So it isn't about unprofitable. Apple isn't out to sell everything to everybody. So they can choose to just do profitable subset of the market. ( they don't need , or even want, "loss leaders" products).
Where Apple doesn't want to be in zones like the 27" 4K HDR monitor market. Which is just fine for a regular apps usage that a $1,699-1,799 iMac might have been applied to previously. Now that can be address by a Mini , Mini Pro , Mac Studio , or laptop plus a decent $400-700 monitor and do solid work with it. The quality of an < $999 monitor has gotten better over the last 6-9 years. It is a zone where Apple doesn't want to go, but many users will.
Some folks don't want a 3rd party display. Apple has two options that are not cheap ( not 'thin' margins). But like when they herded lots of folks into buying an all-in-one when they didn't want one ... this time herding folks into Studio Display when perhaps don't want one. There is always some subgroup faction that Apple is leaving out. ( affordable xMac box with slots , $3K Mac Pro faction , used to be Mini with a decent GPU, etc. etc. )
The other general issue is that Apple only wants to do a relatively small number of Macs ( more time spent on fewer products then their competitors ). If 'had to' do a Studio Display then not necessarily resources around to do a large iMac. ( but Apple is rich they can over staff every project ... well that isn't how they got rich. )