Nothing seems to work still.
Are you sure you removed all the "pkgs"? Though CommandlineTools is one pkg file, it installs many Mac OS X packages (I think it is what they call an mpkg). On one old installation of Xcode 12.4, my system has all these:
com.apple.pkg.CLTools_Executables
com.apple.pkg.CLTools_SDK_macOS1015
com.apple.pkg.CLTools_SDK_macOS110
com.apple.pkg.CLTools_macOS_SDK
com.apple.pkg.DevSDK
com.apple.pkg.DevSDK_OSX1010
com.apple.pkg.DevSDK_OSX1011
com.apple.pkg.DevSDK_OSX1012
com.apple.pkg.DevSDK_OSX109
com.apple.pkg.DevSDK_macOS1013_Public
com.apple.pkg.XcodeSystemResources
com.apple.pkg.macOS_SDK_headers_for_macOS_10.14
Ideally there would be a group or something that would let you uninstall as a bundle but unfortunately that doesn't seem to be the case.
If all files for CommandlineTools were installed under /Library/Developer then "pkgutil --file-info /Library/Developer" would list all packages that went under that subfolder but that heuristic doesn't work for pkgs like "com.apple.pkg.macOS_SDK_headers_for_macOS_10.14" only seem to exist to create dummy files in private/tmp/...
In the end I had to manually review the output of "pkgutil --pkgs" to make that list.
You may also have to run "xcode-select --reset" after all this. Never tried to remove Xcode tools so never tried running that...
Unfortunately MacOS X's package management is a bit of a mess. The pieces appear to be there but it's unfinished relative to what I've seen in other systems even decades ago. Ideally in my mind there would be an algebra such that installing A + B + C - C would give you the exact same system as installing A + B (okay not an real algebra in that A + B + B should not give you A + 2B...).