Well, the OS can be more efficient in writing to disk if it can wait a bit before doing so, because it can line up writes in a more sequential manner. This is the prime reason write caching improves performance (beyond lessening the wait for the user) when using spinning rust disks: those disks write much faster when doing so sequentially as opposed to randomly. (Even when writing a single large file, fragmentation can cause the writes to be in random locations.) The downside to this, is that the write "completes", so far as the user is concerned, well before the data lands on the disk. So if the user thinks, "it's done", and turns off the computer (by turning off the power) or removes the disk, the result can be a corrupted filesystem because in reality the computer was still writing to the disk. And depending on what it was writing the result can be anywhere between an incomplete file and an unusable filesystem.
Remember way back when, when computers couldn't shut themselves off and instead told you 'it is now safe to turn off your computer'? This was why.
Now, the above doesn't really apply to USB sticks much because there is little (some, but not much) gain in writing sequentially. So in that case write caching is done to improve the user experience by speeding up the user-facing action. There is no technical reason in this case, it's just treating a USB stick as any other storage device and caching the writes. Write-caching is the norm.
