Yes, that is the core of the problem. Apparently, Apple only lets you playlist songs that are in your library. So if you add a song to your playlist, it is added to your library.
Forever.
Not quite sure yet if that is a bug or a feature, something to love or something to hate. As I said in another post, I do create some "silly" playlists (like the top 50 songs of a specific year) for fun and nostalgia, but it doesn't mean that I want all of these artists to appear in my library. In fact, it is kinda pointless to even have a library now in the first place. A library is really a pre-streaming kind of concept. Now my library is "all songs on iTunes".