That is not how you exit/quit applications, you exit applications via CMD+Q, File > Quit or right-click on the application's Dock icon and select Quit.
Though closing a window via the X button shouldn't take that long either, but if you have many tabs or documents open, depending on the specs of your Mac, it can take a second or two.