I also found the APP3 noise floor to be quite high. The white noise in quiet to moderately-quiet environments was very obvious and distracting to me – this was most obvious in Noise Cancelling Mode, but also clearly noticed in Adaptive & Transparency mode. Switching to Off Mode gets rid of the white noise entirely but leaves you with bog standard, unremarkable earbuds, so, not a permanent fix of course.
I visited the Apple Store and all diagnostics were normal, but the Specialist advised that Apple had directed them to exchange AAP3 for any users who reported the white noise, even if their diagnostics were normal – makes me think Apple are aware of the issue but haven't yet pinned down the cause... In any case, my second pair of APP3 had the exact same problem.
For now, I've returned them and will stick with my APP2 which have barely perceptible white noise, and their ANC is still excellent. Hopefully the APP3 noise floor is fixed in the future.