I deal with quite a few ~50GB files (each files 50GB) in the last few days, the empty trash is almost instantaneous, however, it will take minutes for Finder to show up the correct empty space available on the APFS SSD (after empty trash).
Couldn't tell the exact reason, but I think only those downloaded files are affected. If I duplicated multiple copies of local file, I didn't get such problem then.