I've also experiencing this for several months on my 16 inch M1 Pro MacBook Pro connected to a Plugable TB dock and 2 external displays. On MacOS 13.4.1
I have no other profile, no FileVault. I have 5 desktops on my MBP display and each external display as its own desktop. So I have 7 different wallpapers in total at a time. The bonkers thing is when I boot up it will actually show me the wallpaper I've set for my first desktop (laptop display) at the login screen.
After I enter my password and login both external displays immediately goes to the default Ventura wallpaper. 4 of 5 desktops on my laptop display will change to the default wallpaper, except for a random desktop like #2 or #3 that would still have my custom one. It may be the desktop I was on when I shut my laptop down.
I had Top Notch installed and turned have that off but no difference. I moved my Wallpaper folder out of the Documents folder into its own folder under my home user folder but also no difference. I also think I tried changing the wallpaper via right click Finder instead of going into Settings but I'm not sure 100% on that, I'll make sure I do it for all 7 desktop displays.
*Update* Did what OP did. I went to each virtual desktop on my laptop display and external displays, opened Finder and went to my custom wallpaper folder and picked the photo, right-click, Set as Wallpaper. It seems to have stuck as I've restarted twice over the weekend and I'm still good for all my 7 desktops wallpapers.