This doesn’t smell right to me. Youtube plays just fine off the YouTube webpage, so what’s the user benefit in the app using different codecs that are incompatible with PiP? And the idea that Google is unable to ’fundamentally rewrite’ it’s app to use an iOS video playback layer is a stretch. There are plenty of apps with far less financial backing than YouTube that manage to play back video just fine without any evidence that the playback framework is intolerably buggy.
It‘s far more likely this simply a business decision to go with one framework because it supports certain business objectives the Apple framework doesn’t. And we all know playing videos isn”t YouTube’s ultimate business objective. I think you’ve hit upon the truth somewhat though. PiP doesn’t allow custom views, so when a PiP YouTube video ends, you don’t get all the recommendations for more videos. You just get an X button to quit the player, and by extension quit YouTube.
So I have some first hand knowledge into how the YouTube app is build. I've also contributed to VLC on iOS as well in years past.
The YouTube app isn't some random basic video player and the content it's playing isn't some perfectly uniform type of content like you get from say a Hulu or HBO. It's a complex app that is willing to do more internally than just use existing high level APIs from the system to improve the experience. The tech behind video on the web is core to YT and they have a lot of highly skilled engineers between YT and Google itself. The same was true of VLC when I worked on it.
That means implementing things on your own instead of using the high level system APIs. There is upsides and downsides to doing that and it's a cost benefit analysis. Downside is that you add a lot of complexity and man hour overhead keeping up your implementation instead of using the free one included and updated for you in the system but the upside is that you full control it which gives you flexibility to do more than what the system APIs provide and fix bugs without waiting for system updates. Entire features are not supported in the system API that YouTube and VLC want, or if they are, they are not supported across all devices. Shipping a built-in decoder in the app gives you the ability to handle things like this on its own though.
That includes supporting novel codecs that apple doesn't care for like VP9 (which it seems iOS 14 is finally adding support for into the system frameworks which means the Apple TV can finally play 4k video since all 4K YouTube videos are only VP9), even on older devices (AVKit, AVFoundation, etc only give you the latest version of things on new devices and OSes). Bundling it in the app allows them to support it on older hardware too. Then you have DRM on some content where YT has it's own DRM system different from the content protection system apple pushes.
Sure, if the iOS app was written today and wasn't born into a world where the iOS high level apis weren't pretty terrible I think the YouTube app may have used those APIs but given the existing investment it's unlikely they will switch. Hell, iOS didn't start to get better video player framework APIs until after iOS 8 when they moved off the MediaPlayer framework for AVKit and the YouTube app is much older than that.
It was a silly move by Apple to couple PIP to AVKit with AVPictureInPictureController instead making it so that any view couldn't be hosted from any app via IOSurface. There are some nasty hacks YT could do to maybe support it. One idea is when switching to PIP the app could open AVKIt player and reload the content in that player using the system framework then (which means blowing up buffers already held by the YT app and having to maybe fetch a video in a format supported by the system which codecs and that may be lower quality) and then trying to sync up the progress to that. Let's say though you downloaded offline a video in the YT app and it's stored in VP9. AVKit is unlikely to play it. Lots of potential edge cases and bugs.
To your other points, the reason it works on mobile safari is because YT will transcode the video into a number of supported codecs so that they are supported in different browsers (WebM, Mpeg2, H264, H265, VP9 etc) and encode the video at different bit rates. They will host up whatever they think will work best for mobile safari but that might be the best looking video at the best bitrate and it might not upgrade to the best bitrate while it's playing in the same way or as well as the native YT app can. High res content will only work in YT app as well since 4K requires VP9.
Go try YouTube on an Android device with PIP there today. I don't know what you mean by showing recommendations at the end and their business model because it works pretty well on Android without anything more than the video in the PIP so obviously not something that matters to them like you claim. Also PIP even on iPad doesn't stop you from auto playing another video.
I think you are way off and don't know what you are talking about.