When I open finder directly after I have logged into the desktop and open Finder it take about 2 second before it show up. If I close the finder window and then open it again there is no delay. Is t just me? I running macOS Big Sur 11.1 beta.
Could be you are clicking too soon after login. Once it is opened. It's in RAM and will open faster. Until it gets pushed out of RAM. If enough apps are opened.